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A BSTSACT 


This  thesis  investigates  the  trends  of  thought  and 
actual  practices  of  commercial  computer  companies  in  the 
area  of  software  quality  assurance.  This  is  done  to  see  if 
any  of  these  practices  could  be  utilized  in  the  Fleet 
Material  Support  Office  (PSSO)  environment.  This  was  accom¬ 
plished  by  personal  interview  of  software  quality  assurance 
personnel  in  a  few  randomly  selected  computer  companies  and 
comparing  thier  quality  assurance  programs  to  that  of  ?«S0. 
The  following  companies  were  selected: 

1.  International  Business  Machines  (IBB)  Corporation, 

2.  TRW  Incorporated, 

3.  Hewlett  Packard  Company, 

4.  Amdahl  Corporation, 

5.  Software  Research  Associates  (SPA). 

Results  indicate  that  the  greatest  differences  between 
the  commerical  world  and  the  FBSO  environment  are  in  manage¬ 
ment's  view  of  what  role  or  function  a  quality  assurance 
group  should  take,  staff  as  compared  to  line,  and  this 
group's  interface  with  the  software  design  and  development 
personnel. 
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I.  INTRODUCTION 


A.  THE  PROBLEM 

Software  quality  assurance  is  the  planned  and  systematic 
actions  required  to  provide  adequate  confidence  that  soft¬ 
ware  products  conform  to  standards  established  by  the 
company  developing  the  product  and  the  contractual  require¬ 
ments  provided  by  the  customer  [Hef.  1].  This  phenomenon 
crosses  all  customer  boundaries:  commercial,  industrial, 
military,  o-.her  governments;  and  crosses  different  applica¬ 
tion  types:  operating  systems,  information  systems,  process 
control,  command  and  control,  communication,  business 
systems,  etc  [Ref.  2]. 

In  the  United  States  Havy  (tISUi  ,  the  office  in  charge  of 
design,  development  and  life  cycle  maintenance  of  the  supply 
system  computer  network  is  the  Fleet  Material  Support  office 
(FMSO)  in  Mechanicsburg,  Pennsylvania.  Or.  29  October  1981, 
FMSO's  Commanding  Officer  established  a  quality  program  task 
group  which  consisted  of  Automatic  Data  Processing  (ADP) 
technical  personnel  from  each  of  its  Central  Design  Agency 
(CD A)  departments  and  supporting  departments.  Its  purpose 
was  to  consider  quality  in  a  broad  sense  as  it  related  to 
the  ADP  system  development  process  and  to  outline  a  general 
plan  for  a  viable  and  continuing  quality  program.  The 
group’s  main  objectives  were  to  provide  recommendations  that 
would  improve  the  quality  of  FMSO  products,  account  for  this 
quality  process  and  sustain  it  throughout  the  product's  lif® 
cycle.  The  conclusions  of  th®  task  group  were: 

1.  Quality  improvement  was  possible  in  the  FMSO 
er.viror.raen- . 
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2.  Qualify  accountability  was  required  and  was  becoming 
increasingly  important.  Correctly  performed,  measure¬ 
ment  would  result  in  an  effective  and  accountable 
quality  process. 

3.  The  ability  to  sustain  acceptable  levels  of  qualify  in 
an  environment  of  changing  .  technology  can  be  accommo¬ 
dated  through  the  iterative  accounting  and  analysis  of 
productivity  and  inventory  characteristics.  *  Ref .  3] 

During  this  same  time  period,  two  other  special  pro-jects 
were  receiving  major  attention  by  FMSC.  Dr.e  is  the 
Resolicit at  ion  project  which  identifies  the  computer 
requirements  at  the  two  Inventory  Control  Points  (HP's)  in 
the  1980's  and  beyond,  taking  into  account  both  the  near 
saturation  of  the  present  Dnivac  491  computers  at  the  ICP's 
and  their  obsolescence.  The  other  project,  called 
"Resvstemization, "  is  also  a  massive  undertaking  as  it  will 
eventually  result  in  new  software  or  computer  programs  for 
the  ICP's.  Talks  between  the  author  and  FSSD's  Commanding 
Officer  indicate  that  this  project  *  3ef .  4]  gives  F!lSO  more 
incentive  to  fake  a  serious  look  at  its  quality  assurance 
program. 

3.  PURPOSE 

The  purpose  of  this  thesis  is  to  Investigate  the  methods 
used  by  large  commercial  computer  companies  in  the  area  of 
software  qualify  assurance.  The  primary  objective  is  to  see 
if  any  of  these  practices  can  oe  utilized  ip.  FMSO's 
env  ir  onment . 
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C.  METHODOLOGY 


The  procedure  used  to  accomplish  the  thesis  objective 
was  to  interview  personnel  from  the  various  computer  compa¬ 
nies.  The  following  companies'  personnel  were  interviewed: 

1.  International  Business  Machines  (IBM)  Corporation, 

2.  TRW  Incorporated, 

3.  Hewlett  Packard  Company, 

4.  Amdahl  Corporation, 

5.  Software  Research  Associates  (5P.A). 

These  companies  were  chosen  because  they  are  located  near 
the  Naval  Postgraduate  School  in  Monterey,  CA  and  they  give 
a  broad  view  of  the  computer  software  industry.  The 
following  questions  were  asked  at  the  interview: 

1.  Where  does  the  quality  assurance  group  fit  into  the 
organization? 

2.  What  type  of  author! ty/power  loss  the  quality  assur¬ 
ance  group  have  over  the  software  product? 

3.  What  qualifications  do  the  people  in  the  quality 
assurance  group  have? 

4.  How  does  the  quality  assurance  group  interface  with 
the  design/developmea t  group? 

5.  What  tools,  methodologies,  or  techniques  does  the 
quality  assurance  group  use  to  do  their  job? 

6.  Are  historical  records  kept  of  problems  with  software 
products  after  their  release  aid  who  in  the  company's 
organization  keeps  them? 

7.  Who  handles  problems  with  software  after  release,  and 
how  are  such  problems  handled? 

3.  If  a  brand  new  product  is  designed,  who  in  the  compa¬ 
ny's  organization  trains  the  customer  on  this  product? 
The  data  was  then  compared  with  existing  practices  at  FMSO 
and  conclusions  and  recommendations  when  rendered. 


D.  STR  tJCTURE  OP  THE  THESIS 

Chapter  I,  the  introduction  to  the  thesis,  presents  the 
thesis  objective  and  methodology.  Chapter  II  presents  a 
general  literature  review  of  the  problem  of  quality  assur¬ 
ance  and  the  factors  that  are  taken  into  consideration  when 
defining  it.  Chapter  III  addresses  the  FMS  environment  and 
its  process  of  quality  control.  Thapter  IV  presents  the 
interviews  conducted  with  the  personnel  of  the  five  computer 
organizations  as  to  their  software  quality  assurance  organi¬ 
zations  and  how  they  work.  The  final  chapter  offers  a 
summary  of  these  interviews  and  provides  reccaaendations  or. 
how  these  ideas  might  be  applied  at  P13D. 
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II 


•  SURVEY  QP  LITERATURE 

Chapter  II  deals  with  the  problem  of  software  quality 
assurance.  After  a  computer  search  to  find  current 
literature  on  this  subject,  it  was  discovered  that  all 
authors  of  these  writings  failed  to  agree  on  the  definition 
of  pertinent  terms.  In  order  to  define  the  terms  relevant 
to  this  thesis,  the  author  presents  the  following 
def in it ions. 

A.  DEFINITIONS 

Software  is  a  set  of  coded  instructions  which  are 
supplied  *ro  and  operate  with  the  computer  hardware  to  cause 
the  hardware  to  perform  the  functions  defined  in  the 
instructions.  [Ref.  5] 

A  system,  as  defined  by  the  Fleet  Material  Supper* 
Office,  is  an  organized  set  of  Automatic  Data  Processing 
(ADP)  hardware,  environmental  and  application  software,  and 
documented  procedures  designed  to  automate  the  basic  manage¬ 
ment  and  operating  processes  for  a  customer  sit?  or  group  of 
customer  sites  with  common  mission  responsibilities 
[Ref.  6].  "Documented  procedures,"  as  used  above,  refers  to 
the  applicable  ADP-celated  and  non- ADP-related  procedures 
established  to  support  the  hardware  and  software  aspects  of 
the  system,  e.g.  the  computer  operation  manual  and  the  users 
manual  [Ref.  7]. 

Qualify  assurance  of  hardware  has  been  successfully 
accomplished  for  many  years,  but  there  are  major  differences 
between  hardware  and  software: 

1.  Software  development  specifications  are  usually  not  as 
specific  as  those  for  hardware.  Precise  sounding 
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tens  with  unspecified  definitions  such  as  "optima" 
or  "99.9  percent  reliable"  are  used  whirl  are  poten¬ 
tial  seeds  of  dissension  or  lawsuits  once  the  software 
is  produced. 

2.  Software  product  (built-to)  specifications  are  usually 
less  rigorous. 

3.  The  software  development  process  is  also  the  produc¬ 
tion  process  because  there  are  no  bread  boards,  brass 
beards,  phototypes  or  pre-production  models  no  use. 

4.  The  production  of  software  (code)  is  neither  a  fully 
constrained  nor  a  uniquely  defined  process. 

5.  The  software  product  itself  (code)  is  essentially  an 
intangible  substance  with  fora,  content,  and  functions 
manifested  via  images. 

5.  Software  problem  fires  always  result  in  a  product 
configuration  change.  (Bef.  9] 

A  basic  software  development  process  is  shown  in  Figure 
2.1.  Corporate  analyses  of  life-cycle  cost  have  shown  that 
the  cost  of  maintenance  and  redesign  exceed  the  cos4:  of 
initial  development  and  tnat  the  cost  of  fixing  errors  after 
the  software  is  operational  is  up  to  33  times  greater  than 
for  correcting  errors  during  system  testing.  Figure  2.2 
shows  a  summary  of  experience  at  International  Business 
Machines,  General  Telecommunications  Equipment  (GTE),  and 
TPM  on  the  relative  cost  of  correcting  software  errors  as  a 
function  of  the  phase  in  which  they  are  corrected.  Figure 
2.2  suggests  that  it  pays  to  iavest  in  one-man  hour 
searching  for  errors  during  the  early  stages  of  development 
than  to  spend  100-tnan  hours  correcting  errors  after  the 
system  is  in  operation.  [Bef.  9] 
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ENT  PROCESS 


FIGURE  2.1  Software  Development  Process 

SOURCE:  Mr.  Kenehan's  Presentation  at  Internal  Meeting  in  TRW 
December,  1981 
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.2  The  Price  of  Procrastination  in  Error  Detection 

Dr.  Barry  W.  Boehm's  Article  on  Software  Engineering 
1  June  1981 


B.  QUALITY  PACTOHS  AND  CRITERIA 


Specific  factors  contribute  to  the  quality  of  software. 
Eleven  cf  these  factors  are  defined  in  Figure  2.3  The 
rationale  [Ref.  10]  behind  the  choice  of  these  is  one  of 
utility;  each  factor  identified  could  be  applied  to  a 
production  environment.  The  interaction  of  support  groups 
within  an  operational  en/ironment  involves  three  distinct 
activities:  product  operation,  product  revision,  and 
product  transition.  Figure  2.4  shows  a  conceptual  scheme 
with  these  three  activities  and  some  related  questions  which 
involve  the  quality  factors  (Ref.  11], 

These  quality  factors  can  be  further  broken  down  info 
criteria  which  could  be  used  for  other  purposes.  Firs*,  a 
set  of  criteria  for  each  factor  further  defines  the  factor. 
Second,  the  criteria  which  affect  more  *har.  one  factor  help 
describe  the  relationships  between  the  factors,  and  the 
criteria  establish  a  working  hierarchical  framework  for 
factors  in  software  quality.  These  criteria  are  defined  and 
their  relations  to  factors  are  shown  in  Pigures  2.5  and  2.6. 
Lastly,  with  the  use  [Ref.  12]  of  these  factors  and  their 
criteria,  a  possible  numerical  value  may  be  added  to  help 
forecast  the  quality  of  the  software  during  its  development 
cycle.  This  is  the  goal  of  software  metrics,  a  tool  used  by 
some  companies  for  this  purpose  [Ref.  13]. 

C.  GENERAL  RESPONSIBILITIES  AND  ORGANIZATION 

Companies  are  finding  that  it  is  advantageous,  from  both 
product  quality  and  cost-effectiveness  standpoints,  to  have 
an  explicit  quality  assurance  activity  on  their  software 
projects  [Ref.  14],  The  tasks  of  mis  activity  are  usually 
tailored  to  the  project  and  depend  on  size  and  scope.  This 
approach  has  proved  effective  in  ensuring  ‘hat  th»  projec* 
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CORRECTNESS 


RELIABILITY 

SFFICIEJJCY 

IMEGRITY 

USABILITY 

NRINTAJNABILITY 

TESTABILITY 

FTZXIBILITY 

PORTABILITY 

PECSABLITY 

T.TEROPESABILrrY 


Extant  to  which  a  program  satisfies  its  specifications 
and  fulfills  the  user's  mssiai  objectives. 

Extent  to  which  a  progran  can  be  expected  to  perform  its 
intended  fimcticn  with  required  precisian. 

The  amount  of  ecnputing  resources  and  code  required  by 
a  program  to  perform  a  function. 

Extent  to  which  access  to  software  or  data  by  unauthorized 
persons  can  be  controlled. 

Effort  required  to  learn,  operate,  prepare  input,  and 
interpret  output  of  a  program. 

Effort  required  to  locate  and  fix  an  error  in  an  operational 
program. 

Effort  required  to  test  a  program  to  insure  it  performs 
its  intended  fwcticn. 

Effort  required  to  rrodify  an  operational  program. 

Effort  required  to  transfer  a  program  from  one  hardware 
aonfiguraticn  and/or  software  systsn  environment  to  another. 
Extant  to  'which  a  progran  can  be  used  on  other  applications 
related  to  the  packaging  and  scope  of  the  f-jicticns  that 
programs  perform. 

Effort  required  to  couple  one  system  with  another. 


FIGURE  2.3  Definition  of  Software  Quality  Factors 

SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A  Survey 


FIGURE  2.4  Allocation  of  Software  Quality  Factors  to  Product  Activity 
SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A  Survey 


FIGURE  2.5  Relationship  of  Criteria  to  Software  Quality 
Factors 

SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A 
Survey 
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FIGURE  2.5  Relationship  of  Criteria  to  Software  Qua! i tv 
Factors  (Contd.) 

SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A 
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DEFINITION 


RELATED 

FACTORS 


TRACEABILITY 

Thbse  attribute*  of  th«  software  that  provide 
a  thread  from  the  requirements  to  the  Imple¬ 
mentation  with  respect  to  the  specific 
development  and  operational  environment. 

Correctness 

COMPLETENESS 

Those  attributes  af  the  software  that 
provide  full  implementation  of  the  functions 
required. 

Correctness 

CONSISTENCY 

Those  attributes  of  the  software  that 
provide  uniform  design  and  implementation 
techniques  and  notation. 

Correctness  | 

Reliability 
Maintainaoil ity  j 

ACCURACY 

Those  attributes  of  the  software  that 
provide  the  required  precision  In  calcula¬ 
tions  and  outputs. 

Rel iaoil ity 

ERROR  TOLERANCE 

Those  attributes  of  the  software  that 
provide  continuity  of  operation  under 
nonnamlnal  conditions. 

Reliability 

i 

SIMPLICITY 

Those  attributes  of  the  software  that 
provide  implementation  of  functions  in  the 
most  understandable  manner.  (Usually 
avoidance  of  practices  which  Increase 
complexity.) 

Reliability  | 
Maintainability  j 
Testability  J 

MODULARITY 

Those  attributes  of  the  software  that 
provide  a  structure  of  highly  independent 
modules- 

Maintainability 

Flexibility 

Testabi 1 ity 

3ortabi 1 1 ty  1 

Reusability  j 

Interoperability 

GENERALITY 

Those  attributes  of  the  software  that 
provide  breadth  to  the  functions  performed. 

Flexibi 1 ity 
Reusability 

EXPANDABILITY 

Those  attributes  of  the  software  that 
provide  for  expansion  of  data  storage 
reouirements  or  computational  functions. 

Flexibi 1 1 ty 

INSTRUMENTATION 

Those  attributes  of  the  software  that 
provide  for  the  measurement  cf  usage  or 
identification  of  errors. 

Testabi i i ty 

l 

:  SEL-- 

OESCRIPT.'ENESS 

1  ’hose  ittr'butss  of  fe  software  fit 

1  provide  explanation  of  the  •moiementat'on 
j  of  a  'unction. 

i 

j 
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FIGURE  2.6  Criteria  Definitions  for  Software  Quality  Factors 


SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A 
Survey 
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CRITERION 

DEFINITION 

RELATED  1 

FACTDRS 

EXECUTION 

EFFICIENCY 

Those  attributes  of  the  software  that 
provide  for  minimum  processing  time. 

Efficiency  ! 

1 

STORAGE 

EFFICIENCY 

Those  attributes  of  the  software  that  1  Efficiency  j 
provide  for  minimum  storage  requirements  |  | 
during  operation.  ;  j 

Those  attributes  pf  the  software  that 
provide  for  control  of  the  access  of 
software  and  data. 

Integrity 

j 

ACCESS  AU0IT 

Those  attributes  of  the  software  that  !  Integrity  j 

provide  for  an  audit  of  the  access  of  I 

software  and  data.  |  | 

OPERABILITY 

-  1  I 

Those  attributes  of  the  software  that  1  Jsaoility  | 

determine  operation  and  procedures  con-  J 

cerned  with  the  operation  of  the  software,  j  i 

TRAINING 

Those  attributes  or"  the  software  that  !  Usability 

provide  transition  *rcm  current  operation  1 
or  initial  famili  an  cation.  j 

EDMMJNICATIVENESS 

Those  attributes  of  the  software  that 
provide  useful  inputs  and  outputs  wnicn 
can  be  assimilated. 

Jsaoility 

SOFTWARE  SYSTEM 
INDEPENDENCE 

Those  attributes  of  the  software  that  I  rortaoility 

determine  its  dependency  on  the  software  Peusaoility 

environment  vooerat'.ng  systems,  utilities, 
input/outout  routines,  etc.)  1 

"ACHINE 

INDEPENDENCE 

Those  attributes  of  the  software  that  !ortaoil'ty 
determine  its  dependency  on  the  naraware  Peusaoility 
system.  j 

1  EEWNIC.ATICNS 
IDMYCNAL IT7 

Those  attributes  of  the  software  that  1  Inzarsser aoi'^t/ 

provide  the  use  ot  standard  protocols 
ano  inter-ace  -lupines. 

data  •::m.'«cnal:ty 

'  .  1 

.hose  attributes  of  :ne  software  that  ;  .nterooeraci * * tv 

provide  the  use  of  stanoaro  data  resre-  1  | 

sentations.  . 

i - 

Those  uttr-  cures  pf  t_e  software  tout  (  "a  :  r,  ta :  nap : "  •  t  • 
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FIGURE  2.6  Criteria  Definitions  for  Software  Quality  Factors 
(Contd.) 


SOURCE:  Macabe's  Book  on  Software  Quality  Assurance  -  A  Survey 
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is  responsive  to  the  quality  requir eatents  of  the  customer 
and  to  the  particular  system  application.  The  general 
responsibilities  of  such  an  activity  include: 

1.  Planning 

a)  Preparation  of  the  quality  assurance  plan  staging 
duties,  responsibilities,  and  schedule. 

b)  Project  and  customer  interfaces. 

c)  Resource  management. 

d)  Subcor.tractor/supp  lier  management. 

2.  Policy,  Practice  and  Procedure  Development 

a)  Preparation  of  standards  manuals  for  ail  phases  of 
the  software  production,  including  requirements 
design,  coding,  and  test  tailored  to  specific 
project  requirements. 

b)  Problem  reporting  and  analyses. 

3.  Software  Quality  Assurance  Aids  Development 

a)  Adaptation  of  existing  tools  or  methods. 

b)  Development  of  manual  and  automated  procedures. 

c)  Keeping  abreast  of  new  ani  "state  of  the  arin  aids. 

4.  Audits 

a)  Review  of  project  procedures  and  documentation  for 
compliance  to  standards. 

b)  Participation  in  interim  reviews. 

c)  Participation  in  customer  audios  of  the  project. 

d)  Quality  assurance  inspections. 

5.  Test  Surveillance 

a)  Participation  in  the  testing  phase. 

b)  Reporting  of  software  problems. 

c)  Analysis  of  error  causes  and  assurance  of  correc¬ 
tive  action. 

Records  Retention 

a)  Quality  assurance  records  management. 
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b)  Retention  of  problem  reports,  *est  rases,  test 
data,  logs  of  quality  assurance  reviews. 

c)  Insure  proper  doca mentation. 

7.  Physical  Media  Control 

a)  Inspection  of  dish,  tapes,  cards,  and  ether 
program-retaining  media  -  verification  at  all  •‘rimes 
of  physical  transmittal  or  retention. 

b)  Protection  from  mishaniLing  or  altering  by 
environment.  [Ref.  15] 

The  classical  quality  assurance  group  role  or  interface 
with  the  development  cycle  usually  roses  at  the  end  of  th* 
development  cycle  when  testing  starts.  Their  job  is  to 
dissect  the  problem,  find  errors,  test  for  the  environment 
in  which  the  software  product  is  to  be  used  in  and  nctifv 
the  developers  of  faults.  This  sometimes  produces  an  adver¬ 
sary  relationship  between  the  groups,  destroying  any  cooper¬ 
ation  or  aid  one  might  give  the  other.  The  autonomy  of  the 
quality  assurance  group  is  also  imperative  for  achieving  any 
type  of  success.  [Hef.  15] 

In  software  production  environments  today,  the  quality 
assurance  group's  intention  is  to  work  with  the  development 
side  of  the  house  throughout  the  development  cycle.  They 
view  themselves  as  a  tool  or  aid  to  the  management  of  the 
development  process,  informing  the  manager  of  problems  thev 
see  as  a  hinderance  to  the  schedule  or  quality  of  the 
product  under  development.  The  autonomy  of  this  group  is 
still  important.  [Ref.  17] 

D.  SUMMARY 

This  chapter  has  listed  the  questions  which  must  be 
answered  about  the  software  product  before  the  duties  of  the 
quality  assurance  group  can  be  delineated.  Along  with  these 
questions,  the  exact  role  of  the  qaaLLty  assurance  group  and 
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its  interfaces  with  the  development  group  may  be  viewed 
differently,  depending  on  the  character  of  the  organization 
itself.  The  following  chapters  define  the  purpose  and  envi¬ 
ronments  of  the  quality  assurance  organizations  under 
consideration  and  explain  their  process  of  quality 
assurance. 
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III.  IhlZl  hAT ERIAL  SJPPDRT  OFFICE 


The  purpose  of  this  chapter  is  to  describe  the  FMSO 
environment  and  the  process  of  software  quality  assurance  in 
this  organization.  The  following  references  were  used: 

1.  Fleet  Material  Support  Office  Organizational  Manual 

2.  Fleet  Material  Support  Office  Central  Design  Agency 
(CDA)  Management  Handbook,  1  January  1981, 

3.  Fleet  Material  Support  Office  Internal  Instruction 

5230. 20A  CDA  Development  Handbook,  1  December  1979, 

4.  Fleet  Material  Support  Office  Internal  Instruction 

5230.12  Quality  Assurance  Program,  17  May  1973, 

5.  Fleet  Material  Support  Office  Quality  Program  Task 

Group  Report,  31  January  1982, 

5.  The  Navi  SuEEil  Co tds  Newsletter,  January  1982, 
Special  Issue  “Celebrating  FMSD’s  20th  Anniversary.” 

A.  HISTORY 

Established  in  1962,  FMSO  was  originally  chartered  to 
provide  central  management  for  the  retail  portion  of  the 
Navy  Stock  Fund  (NSF) .  It  was  also  used  to  obtain  and  stock 
supplies  from  other  services.  It  also  catalogued  data  for 
supply  system  performance  analysis  and  evaluation. 
Originally  this  organization  consisted  of  five  officers  and 
56  civilian  employees,  but  today  it  has  grown  to  more  than 
33  military  personnel  and  over  1,303  civilians. 

The  main  reason  for  the  organization’s  growth  has  beer, 
its  increase  in  responsi bilities.  The  first  addition 
occured  in  1965  when  •‘■he  Central  Design  Agency  function  was 
incorporated  info  its  mission  areas.  This  function  involves 
the  design,  development  ar.d  life  cycle  maintenance  of 
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programs  used  in  computer  systems.  This  initial  designation 
was  limited  to  computer  systems  used  in  supply  and  financial 
operations  at  various  field  activities. 

In  1973,  FMSO's  direct  relationship  with  the  fleet  was 
increased  with  the  assignment  of  the  3a  program.  This  func¬ 
tion  was  reassigned  to  the  Navy  maintenance  support  office 
in  1978.  In  1977,  two  additional  increases  in  FMSO's 
mission  area  occurred.  The  financial  systems  role  was 
significantly  expanded  with  the  assignment  of  ZD  A  responsi¬ 
bility  for  financial  systems  utilied  by  headquarters  activ¬ 
ities  in  Washington,  D.Z.,  such  as  the  Naval  Material 
Command  and  various  systeas  commands.  The  other  expansion 
was  the  result  of  FMSO's  designation  as  the  CDA  for  the 
Trident  Logistics  Data  System,  whicn  added  submarine  inter¬ 
mediate  level  maintenance  to  PMSO's  CD  A  mission.  The  most 
recent  addition  to  their  mission  area  occurred  in  1978  with 
the  responsibility  assignment  of  the  prototype  development 
for  the  Naval  Aviation  Logistics  Command  Management 
Information  System  (NALCOMIS). 

Approximately  80%  of  FMSO's  work  force  is  engaged  in  CDA 
activities.  A  significant  portion  of  that  effort  is 
expended  in  four  Uniform  Automatic  Data  Processing  Systems 
(UADPS)  :  the  Uniform  ADP  System  for  Inventory  Control 
Points  (UICP) ,  the  Uniform  ADP  System  for  Stock  Points 
(UADPS-SF) ,  the  Level  II/III  system,  and  the  Disk  Oriented 
Supply  System  (DOSS).  A  list  of  the  user  sites  for  each 
system  appears  in  Figure  3.1. 

8.  ORGANIZATIONAL  STRUCTURE 

Figure  3.2  shows  the  organizational  structure  of  FMSO. 
Two  departments  carry  out  ill  of  the  staff  functions  such  as 
resource  management,  operation  and  maintenance  Navy  budg¬ 
eting,  plar.ning/admin  ist  rat  ion,  production  support,  project 
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FIGURE  3.1  Index  of  FMSO  CDA  Systems  Users 
SOURCE:  CDA  Management  Handbook 


3.2  FMSO  Organization 
FMSO  Organization  Manual 


control,  standards  develop ment,  data  base  adiinistration, 
training  and  ADP  operations  support.  These  are  the 
Comptroller  Department  (Coda  9 1 1  and  the  Management 
Department  (Code  92).  The  software  quality  control  branch 
is  in  the  planning  division  of  code  92.  (Figure  3.3)  The 
Comptroller  Department  also  performs  external  missions 
including  stock  fund  budgeting  and  direct  fleet  support 
functions. 

The  Operations  Analysis  Department  (Code  93)  is  the 
Naval  Supply  Systems  Command's  (MAVSJPI  principal  agent  for 
conducting  analysis  in  logistics  management.  This  depart¬ 
ment  is  made  up  of  operations  research  analyses  and  mathema¬ 
ticians  who  use  various  mathematical,  statistical  and 
economic  analysis  techniques  to  study  and  improve  the 
procurement,  financial  and  inventory  management  functions 
throughout  the  United  States  Navy.  These  services  are  also 
provided  for  all  NAVSUP  activities,  the  fleet.  Chief  of 
Naval  Operations  and  Chief  of  Naval  Material  offices,  other 
systems  commands  and  various  project  managers. 

C.  THE  CDA 

"A  central  design  agency  is  defined  as  a  single  organi¬ 
zation  which  designs,  develops,  implements  and  maintains 
automated  data  processing  systems  in  support  of  multiple 
operating  sites."  [Ref.  13]  The  five  FMSO  CDA  production 
departments  (Code  94  through  93)  are  the  line  organizations 
which  are  directly  responsible  for  tie  development  and  main¬ 
tenance  of  standard  ADP  systems.  The  personnel  in  these 
departments  are  functional  systems  designers,  computer 
systems  analysts,  computer  specialists,  and  computer 
programmers.  Their  wor  k,  development  and  dcciier*  a*  ior.  of 
these  programs,  is  the  major  product  of  the  CDA. 
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Code  92  Organization 
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Three  basic  principles  necessitate  the  existence  of  this 
type  of  production  organization  and  directly  impinge  or.  its 
effectiveness. 


1.  There  must  be  a  potential  group  of  customer  sites 
which  perform  a  mission  of  functional  similarity  and  operate 
with  business  volume  of  a  magnitude  sufficient  to  justify 
acquisition  and  operation  of  automated  systems. 

2.  The  functional  similarity  of  the  individual  sites 
within  the  customer  group  must  be  complete  enough  to  permit 
a  degree  of  system  standardization  by  which  the  single 
product  of  the  design  agency  can  adequately  support  the 
needs  of  multiple  users,  thus  the  cost  of  system  development 
and  maintenance  can  be  defrayed  by  the  benefits  obtained  by 
the  many  users.  At  the  sane  time,  a  marked  degree  of  stan¬ 
dardization  and  improved  management  is  obtained. 

3.  The  concentration  of  system  design  and  development 
talent  in  a  CDA  affords  opportunities  for  single  operating 
sites  tc  obtain  development  of  systems  that  they  could  not 
afford  to  develop  themselves. 


The  objectives  of  a  CDA  is  as  follows: 
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-  To  develop  ri gidly-umf  orm  programs 
with  design  options,  alternatives  and 
modularity  which  facilitate  subsequent 
policy/procedural  changes  and  accomodate 
unique  customer  requirements  with  due 
consideration  of  efficiently/flexibility 
trade-offs. 

To  design  and  develop  ADP  systems 
which  will  be  compatible  with  the 
projected  role  of  user  sites  during 
future  years. 

To  participate  in  the  exchange  of 
information  with  other  ODD  desian  agen¬ 
cies  and  to  enhance  systems  effective¬ 
ness  and  personnel  proficiencv. 

-  To  identify  project  resource  require¬ 
ments  in  the  initial  planning  stage  so 
that  sufficient  lead  time  is  provided 
for  timely  acquisition  and  development. 

-  To  prepare  CDA  budgets  which  reflect 
sound  and  integrated  prediction  plans; 
to  allocate  resources  within  the  CD A  in 
accordance  witi  reconciled  budget/pro¬ 
duction  plans. 

-  To  optimize  CDA  organizational  struc¬ 
ture,  staffing  levels,  and  allocation  of 
personnel  resources  in  order  to  insure 
maximum  productivity  on  high  priority 
projects. 

-  To  pursue  oersor.nel  recruitment  and 
training  programs  which  insure  avail¬ 
ability  of  advanced  knowledge  and  skills 
in  logistics,  data  processing,  financial 
management  and  related  disciplines. 

-  To  enhance  CD A  productive  capability 
thorugh  the  use  of  special  tools, 
including  interactive  programming,  data 
base  management,  pre-compilers,  and 
other  available  techniques. 

-  To  employ  the  most  effective  training 
techniques  available  in  order  to  imple¬ 
ment  systems  at  new  user  sites  and 
install  new  aoolications  at  existing 
user  sites:  to'  conduct  a  orogram  of 
field  assistance  which  assures  continued 
proficiency  of  user  sites  in  operations 
supported  by  CDA  systems. 

-  To  utilize  standard  high-level 
programming  languages  to  the  maximum 
extent  feasible  and  to  use  assembly 
languages  only  where  technical  require¬ 
ments  unequivocally  dictate."  [Ref.  19] 


While  all  of  the  CDAs  are  involved  ir.  basically  th 
operation,  they  are  separated  into  logical  functional 
of  support.  Because  of  this  separation,  the  CDAs  do  r. 
serve  the  same  customers.  FMSO  as  a  CDA  is  divided  ir. 
following  areas: 


3« 


e  same 
areas 
ot  all 
to  the 


Environ  Mental  §.l§tegs  03  5  ion  and  Development  Department 
(Code  94)  (Figure  3.4) 

This  department  is  responsible  for  the  design,  development, 
implementation  and  maintenance  of  environmental  systems 
software  in  support  of  NAVSUP-sponsored  ADP  systems, 
including  UADPS  of  stock  points,  0IEP,  the  trident  program, 
the  international  logistics  program,  and  programs  that  are 
assigned.  This  department  also  performs  these  functions  for 
the  systems  maintained  by  the  other  CDA's. 
Telecommunications  networks  sponsored  by  the  '(aval  Supply 
Systems  Command  are  another  area  in  which  code  94  is  respon¬ 
sible  for  the  environmental  systems  software.  This  depart¬ 
ment  is  made  up  of  109  computer  specialists  and  27  people 
who  handle  all  managerial  and  clerical  activities.  Other 
major  projects  either  designed  or  supported  are: 

1.  SPLICE  -  stock  point  logistics  integrated  communica¬ 
tions  environment 

2.  LDC  -  looistics  data  communications 

3.  OLA  -  on-line  autodin 

4.  AOTODIN  II  -  automatic  digital  network 

Stock  Point  Systems  Design  and  Procedures  Department  (Code 
95)  (Figure  3.5) 

This  department's  purpose  is  to  develop  and  maintain  the 
automated  systems  for  Navy  stock  point  support  including 
trident  Logistic  Data  System  (LDSI  ,  NALCOMIS,  Automated 
Heady  Supply  Stores  System  (ARSS5) ,  Tape  Oriented  Supply 
System  (TOSS),  Disk  Oriented  Supply  System  (DOSS),  Electric 
Pcir.t  cf  Sale,  level  II,  Navy  Automated  Ir anspcrtatior. 
Documentation  System  (NA7ADS),  Navy  Automated  Transportation 


Data  System  (NATOS),  Transportation  Operational  Personal 
Property  Standard  System  (TOPS),  Navy  Integrated 
Storage-Tracking  and  Retrieval  System  (NISTARS) ,  Requisition 


Material  Monitoring  and  Expediting  (RMSSE) 


Closed  Loop 
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FIGURE  3.4  Code  94  Organization 
SOURCE:  FMSO  Organization  Manual 
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FIGURE  3.5  Code  95  Organization 
SOURCE:  FMSO  Organization  Manual 
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Aeronautical  Management  Program  (CLASP)  ,  and  Defense 
Warehousing  and  Shipment  Process  (DWASP).  This  department 
also  assists  customers  with  the  implementation  of  c hese  ADP 
systems  through  development  of  training  documentation, 
initial  training  and  installation  assistance,  monitoring  of 
performance  under  operational  conditions  and  follow-on  field 
assistance.  The  department  is  involved  with  approximately 
40  Navy  stock  points  located  around  the  world. 

Inventory  Control  Joints  Design  and  Procedures  Department 
(Code  96)  (Figure  3.6) 

The  purpose  of  this  department  is  to  develop  and  maintain 
the  TCP's  OADPS  design  and  work  on  refinements  to  these 
proqrams  to  carry  out  NAV53?  and  hardware  SYSCDMS  inventory 
control  functions.  Their  principal  customers  are  the  two 
major  Navy  ICPS:  the  Ships  Parts  Control  Center  (SPCC)  and 
the  Aviation  Supply  Office  (ASO) .  This  department  also 
develops  and  maintains  detailed  systems  design  for  trident 
and  ship-support  functions.  It  is  comprised  of  approxi¬ 
mately  250  people  and  is  a  functionally  oriented  department. 
The  Financial  Systems  Design  and  Procedures  Department  (Code 
97)  (Figure  3-7) 

This  organization  is  responsible  for  systems  design,  devel¬ 
opment  implementation  and  maintenance  services  for  headquar¬ 
ters,  Naval  material  command;  Chief  of  Naval  Material 
designated  project  management  offices;  and  other  partici¬ 
pating  headquarters  commands  and  offices.  It  provides 
service  to  both  of  the  major  customer  groups;  inventory 
control  points  and  stock  points  and  ether  activities  under 
the  (JICP  and  tJADPS  programs  in  the  areas  of  financial  inven¬ 
tory  control,  stores  accounting,  disbursing,  plant  property, 
payroll  and  personnel  accounting. 

The  systems  designed  by  this  organization  supports  91% 
of  the  Navy's  financial  inventory  report  requirements,  75% 
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of  the  current  Navy  dollar  resources  under  its  resource 
management  system,  and  63^  of  189,300  civilian  employee 
salaries. 

Code  97  provides  similar  services  to  the  Navy  regional 
finance  centers  and  evaluates  the  performance  and  develop 
such  projects  as  the  Integrated  Disbursing  and  Accounting 
(IDA)  System,  Standard  Accounting  and  Reporting  System 
(STARS)  and  the  Automated  Procurement  and  Accounting  Data 
Entry  (APADS)  System. 

This  department  consists  of  three  military  officers  and 
a  civilian  complement  of  244,  covering  the  full  range  of 
financial  systems  and  data  processing  expertise. 

Ir.t er national  Logistics  Sup po^t  Department  (Code  98)  (Figure 
3.8) 

This  department  is  responsible  for  the  maintenance  and 
continual  enhancement  of  the  Management  Information  System 
for  International  Logistics  (HISIL).  Its  principal  customer 
is  the  Naval  International  Control  officer  (NAKILCD)  which 
utilizes  its  systems  to  provide  services  to  numerous  allied 
navies  and  governments.  The  department  handles  complete 
automation  for  the  Saudi  Arabian's  Navy  supply  system  and 
automation  of  support  systems  (supply,  environmental, 
personnel  and  financial)  for  Kuwait's  Navy.  it  establishes 
training  programs  for  Inited  Stares  Navy  Supply  Corps 
personnel  going  to  Military  Assistance  Advisory  group  (MAAS) 
duty  and  develops  an  advance  base  supply  system  for  overseas 
supply  depots. 

D.  SYSTEM  DESIGN  AND  QUALITY  ASSURANCE  PROCESS 

The  top  down  design  method  is  used  as  the  standard 
approach  for  new  systam/p rogram  development  in  the  FMSO 
environment.  This  approach  is  also  Jcnown  as  stepwise 
refinement,  hierarch ial  design,  levels  of  abstraction  ’ad 
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FIGURE  3.8  Code  98  Organization 
SOURCE:  FMSO  Organization  Manual 


design  by  explosion.  The  aethod  uses  a  breakdown  technique, 
dividing  the  sain  function  into  smaller  subf unctions.  The 
primary  function,  thought  o  f  as  the  central  or  driving  func¬ 
tion,  is  designed  first;  then  stepwise,  this  process  is 
continued  until  the  smallest  functional  unit  of  the  system 
is  specified. 

Because  of  this  breakdown,  the  system  can  be  viewed  as 
modules.  Every  stage  of  the  system  and  program  yields  a 
visible  output.  Each  subsequent  subfunction  which  is 
defined  becomes  a  module  of  code  which,  when  tested,  serves 
to  retest  and  more  thoroughly  test  aLl  higher  level  modules. 

The  use  of  hierarchical  charts  forces  the  design  of  new 
system/programs  in  the  top  down  method.  This  use  of  visual 
diagrams  shows  the  major  functions  and  their  subfunctions 
with  the  emphasis  on  their  subordination  and  not  their 
logical  flow. 

FNSO  personnel  state  that  the  system  designers  focus  on 
what  is  required  and  the  systems  analysis  workers  focus  on 
how  to  achieve  it.  The  system  designer,  working  very 
closely  with  the  system  user,  defiles  what  information  is 
required,  how  it  is  required,  when  it  is  required,  and  for 
whom  it  is  required.  This  helps  tremendously  in  keeping 
this  process  of  development  at  minimum  cost. 

The  system  development  process  is  deliniated  in  FMSO's 
CPU  Management  Handbook.  Appendix  A,  taken  from  the  hand¬ 
book,  shows  the  process. 

During  the  development  process  a  quality  assurance 
checklist  is  required.  Figure  3.3  is  an  example  of  rhs 
checklist. 

On  31  January  1982,  a  qualify  program  task  group  report 
was  published.  In  this  report  were  the  results  received 
from  the  following:  an  internal  survey  taken  from  -.he  CPAs; 
an  examination  of  the  AD?  development  model  and  the  CPA 
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Program/ version_ 
ELEMENT 


QDALITV  ASSURANCE  CHECKLIST 

Data 


SIGNATURE 


DATE 


Scope  of  Release: 

a.  New  program/ cooplete 

— 

rewrite 

b.  Major  modification 

c.  Moderate  revision 

d.  Minor  adjustment 

_ 

2.  Criticality  of  Releaae: 

a.  Mandatory  (HQ.  directed) 

b.  PTR  response 

c.  Solves  serious  program 
deficiencies 

d.  Highly  desirable 
enhancement 

e.  Routine  release 


3-  Urgency  of  Implementation: 

a.  Isnedlate 

b.  Ho  later  than  ______ 

e.  Optional 


4.  Level  of  Testing: 

a.  Local  FMSO  testing  with 
simulated  test  data 

b.  Service  tested  at 


c.  Prototyped/Op  Reviewed 
at  ____________ 

d.  Tested  by  FMSO  with  live 

data  fllaa/transactlons 
fro* _ 


FIGURE  3.9  Quality  Assurance  Checklist 
SOURCE:  FMSO  Quality  Assurance  Program 
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ELEMENT 


SIGNATURE 


DATE 


FIGURE  3  9  Quality  Assurance  Checklist  (Contd.) 
SOURCE:  FMSO  Quality  Assurance  Program 
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El. EM  ENT 


SICNATURE 


DATE 


11.  Satisfies  System/ Subsystem  Speci¬ 
fication  as  Approved 

12.  Satisfies  Program  Specification 

13.  File  Integration/Integrity 
Verified 

14.  System  Intogratlon/Xntegrity 
Verified 

15.  Tested  In  (Simulated/ 

Production)  Environment 

16.  Test  Data  Base  Updated  To 
Ensure  Adequate  "Real  World" 

Cases 

17.  Program  Restart  Capability 
Verified 

18.  Program  Interfaces  with  Software: 

a.  Currently  Implemented 

b.  Hew  Software  Package 

Required  _ 

c.  Scheduled  for  Release 


19.  (Software)  Release  is  Upward 
Compatible  with  Prior  Releases 

20.  Programs  have  boon  developed, 

analyzed,  coded  and  reviewed  at 
critical  steps  utilizing  the 
FMSO  standard  Improved  Pro¬ 
gramming  Techniques,  as  described 
In  FNSOINTINST  _ . 

21.  User  Training  Has  Been  Provided/ 
Is  Mot  Required 

e.  Type  Training  Provided 

b.  Dace  Training  Completed 
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FIGURE  3.9  Quality  Assurance  Checklist  (Contd.) 
SOURCE:  FMSO  Quality  Assurance  Program 
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22.  Standard  data  a 1 affiant  naaas  have 
bean  usad  throughout  tha  program 
coding. 

23.  Remarks/qualif ication/explanation : 


24.  Element  cartification  responsibilities:  sea  item  24.  anclosura  (4) 

for  individual  element  cartification  responsibility. 

25.  QUALITY  ASSURANCE  CHECKLIST  CERTIFICATION. 

Each  of  the  above  quality  assurance  checkpoints  has  been  verified/ 
validated  by  myself  or  by  parsons  under  ay  supervision.  Tha  responses 
given  are  true  and  correct  to  the  best  of  ay  professional  knowledge.  I 
understand  that  individual  quality  assurance  level  is  a  significant 
factor  in  each  annual  performance  rating.  I  certify  that  this  program 
release  has  met  all  FMSO  quality  assurance  tests  and  standards  and  is 
ready  for  release  to  customer  activities. 


Branch  Head  Date 


Division  Head  Date 


Departannt  Head  Data 
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FIGURE  3.9  Quality  Assurance  Checklist  (Contd.) 
SOURCE:  FMSO  Quality  Assurance  Program 
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handbooks;  a  review  of  the  functional  operations  of  the 
quality  control  organization,  research  and  review  cf  the 
technical  libraries  and  publications  dealing  with  software 
quality  assurance  programs,  and  an  external  survey  question¬ 
naire  directed  to  the  FMS3- systems  customer  community. 

The  report  stated  that  the  following  factors  in  the  FMS3 
environment  prejudice  quality  in  varying  degrees: 

1.  Mandated,  multiple  and  dissimilar  hardware 

configurations. 

2.  Onraalistic/inf lexibl e/mandate  project  completion 

dates. 

3.  Ill  defined  or  undocumented  requirements. 

4.  Inadequate  test  facilities. 

5.  Funding  (budget/travel)  constraints. 

5.  Project  prioritization  process. 

7.  Diversity  of  custoaer  activity  in  systeas/processing 

requirments. 

9.  System  changes /controls  edicted  from  agency/system 
command  echelons. 

9.  Federal  procurement  policies  and  regulations. 

The  task  group's  work  experience,  a  review  of  industrial 
literature,  and  the  internal  survey  revealed  that  the 
following  specific  conditions  exist: 

1.  Poorly  Defined  Requirements/Specifications 

a)  FMSO  design  procsd ures/practices  tend  to  be  appli¬ 
cation-oriented  and  at  the  discretion  of  xhe 
developer. 

b)  System  design  and  analysis  knowledge  is  no*  being 
shared  between  or  within  ths  3DAs. 

c)  Formal  review  and  walkthroughs  are  not  beinj 
carried  out  properly  during  system  development. 

d)  There  is  no  visible  interaction  with  customers. 
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e)  System  analysts  are  not  always  required  during  unit 
testing. 

f)  With  the  exception  of  the  program  trouble  report, 
there  is  no  provision  for  soliciting  or  consoli¬ 
dating  customer  feedback  information  on  a  recur¬ 
ring  basis. 

g)  ADP  system  developmental  information  and  experi¬ 
ence  is  not  formally  or  consistently  shared  among 
developmental  organizations. 

h)  A  more  business-like,  comprehensive  policy  and 
procedures  document  is  necessary  for  FMSO/customer 
relationships. 

2.  Unrealistic  Schedules/Estimatei  Completion  Dates 

a)  Mandatory  due  dates  cause  abbreviation  of  quality 
events. 

b)  Completion  date  as  set  by  tie  POA&M  is  usually  "set 
in  stone." 

c)  Project  tracking/status  reporting  and  resource 
accounting  are  not  currently  provided  on  an  inte¬ 
grated  basis  for  project  management. 

d)  There  is  limited  automated  capability  in  the  areas 
of  documentation  preparation,  storage,  assembly, 
packaging  and  distribution. 

3.  Insufficient  Testing  Time/Test  Facilities 

a)  Unreliabiliy  of  hardware  (F1S3)  ,  basically  the  test 
beds,  precludes  estimating  realistic  time  frames 
and  completion  dates. 

b)  There  is  lack  of  uniformity  in  the  assignment  of 
specific  responsibilities  in  program/system 
testing. 

c)  No  uniform  methods  or  procedure  exist  fcr  estab¬ 
lishing  and  maintaining  F133's  test  fil=>s. 
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d)  In  undisciplined  approach  to  program  testing  among 
CD As  is  used. 

e)  Software  engineering  is  not  a  distinct  function. 

'4.  Lack  of  "Stat  e-of-The-Art”  Developmental  Tools  and 
Aids 

5.  Or.ecessary  Paperwork  and  Processes 
E.  CONCLUSION 

As  shown  in  the  system  development  process.  Appendix  A, 
the  quality  assurance  branch  interfaces  with  development 
personnel  in  tracking  of  the  functional  description  and 
system  specifications  and  in  checking  the  product  before 
release  for  compliance  with  standards  and  quality  assurance 
procedures  (check  list) .  All  rests  and  project  reviews  are 
carried  out  by  the  development  personnel  with  the  use  of  the 
quality  assurance  check  list.  The  actual  duties  of  the 
quality  assurance  branch  may  be  viewed  as  only  administra¬ 
tive  in  nature.  The  next  chapter  shows  how  other  quality 
assurance  qroups  function  in  their  organizations. 


53 


IV 


INTERVIEWS 


This  chapter  presents  the  author-conducted  interviews 
with  personnel  of  the  quality  assurance  groups  in  the 
computer  organizations  addressed  in  Chapter  r.  The 
following  questions  were  asked  during  the  interview: 


1 . 

Where  does  the 

organiz  ation? 

quality  assurance 

group  fi* 

into  the 

2  . 

What  type  of 

auth  orit  y/power 

does  the 

quality 

assurance  group  have  over  the  software  product? 

3.  What  qualifications  do  the  people  in  the  quality 
assurance  group  have? 

!» .  How  does  the  quality  assurance  group  interface  with 
the  des ign/deve lopmen t  group? 

5.  What  tools,  methodologies,  or  techniques  does  the 
quality  assurance  group  use  to  do  their  job? 

6.  Are  historical  records  of  problems  with  software 
products  kept  after  the  products'  release,  and  who  in 
the  company's  organization  keeps  them? 

7.  Who  handles  problems  with  software  after  releas®,  and 
how  are  such  problems  handled? 

3.  If  a  brand  new  product  is  designed,  who  in  the 
company's  organization  trains  the  customer  on  this 
product? 

The  reader  is  enjoined  to  compare  the  interviews  with 
the  discussions  in  Chapters  II  and  III. 

A.  HEWLETT  PACKARD 

The  Hewlett  Packard  Company  is  a  major  designer  and 
manufacturer  of  precision  electronic  equipment  for  measure¬ 
ment,  analysis  and  computation.  The  company  makes  mere  than 
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4,000  products  which  are  sold  worldwide  and  have  broad 
application  in  the  fields  of  science,  engineering,  business, 
industry,  medicine  and  education.  Their  four  main  product 
segments  are: 

1.  Electronic  Data  Products  --  computational  products 
including  personal  computing  devices,  desk  top  computers  for 
engineering  and  scientific  applications,  small  business 
computers,  and  larger  computer  systems  for  both  business  and 
technical  needs.  They  also  offer  a  large  selection  of 
application  software  and  have  developed  a  wide  selection  of 
peripheral  equipment  for  use  with  their  computers,  including 
computer  terminals,  disc  memories,  printers  and  plotters. 

2.  Electronic  Test  and  Measurement  Products  --  range 
from  general  purpose  instruments  and  systems  for  electronic 
test  and  measurement  to  specialized  instrumentation  for 
computed  measurements  to  components  and  accessories  such  as 
microwave  semiconductors,  optoelectric  displays,  bar  code 
readers,  and  fiber  optic  systems. 

3.  Medical  Electronic  Equipment  --  family  of  more  than 
303  medical  products  which  are  used  for  diagnosing,  moni¬ 
toring,  and  treating  patients,  and  for  medical  information 
management.  This  equipment  ranges  from  portable  electrocar¬ 
diographs  to  powerful  computer-aided  patient  monitoring  and 
patient  data  management  systems. 

4.  Analytical  Instrumentation  --  Product  family 
includes  gas  and  liquid  cnr omatographs,  mass  spectrometers, 
automatic  fluid  samplers,  analytical  laboratory  data  acqui¬ 
sition  systems,  and  spect r ooho tomet srs .  This  instrumenta¬ 
tion  is  used  for  research,  production,  and  environmental 
applications. 
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1 .  Organization 


Figures  4.1  thru  4.3  show  tie  organizational  struc¬ 
ture  of  the  Hewlett  Packard  Company.  In  the  computer  area, 
there  is  the  technical  computers  group,  of  which  the  Data 
Systems  Division  is  a  part.  The  products  or  quality  assur¬ 
ance  organization  comes  from  this  division.  This  organiza¬ 
tion  is  not  only  responsible  for  software  quality  assurance 
but  also  for  hardware  quality  assurance,  production  support, 
product  reliability,  information  systems,  quality  assurance, 
production  regulation  and  safety,  etc.  The  software  quality 
assurance  engineering  group  is  made  up  of  14  people  who  have 
the  education  and  experience  to  be  program  designers  and 
programmers  themselves,  but  their  job  is  strictly  quality 
assurance.  Their  main  purpose  is  to  work  along  with  the 
product  designers  from  the  research  and  development  group, 
assisting  them  in  designing  a  quality  product.  This  inter¬ 
face  between  designers  and  quality  assurance  people  is  not 
true  for  all  areas  of  Hewlett  Packard  production,  but  the 
company  is  moving  in  that  direction. 

The  quality  assurance  group  does  not  have  absolute 
authority  over  the  product.  Absolute  power  would  m^a n  that 
if  they  thought  the  product  was  not  ready,  it  would  not  be 
released.  They  state  that  their  real  power  lies  in  their 
reputation  and  their  ability  to  persuade.  If  they  predict  a 
failure  and  it  occurs,  the  group’s  credibility  and  reputa¬ 
tion  are  enhanced,  and  the  persuasion  speaks  for  itself. 
The  division  general  manager  makes  the  final  decision  or. 
whether  a  product  is  released,  and  it  is  the  job  of  the 
quality  assurance  personnel,  in  competition  with  design 
people,  to  convince  him/her  that  the  product  is  not  ready. 
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4.<?  Hewlett  Packard  Data  Systems  Division 

Interview  with  Hewlett  Packard  Quality  Assurance  Manager 


2.  fllftliti  issuance  a  nd  Desijai£  Inte^age 


Figure  4.4  shows  the  development  cycle  as  it  is 
perceived  by  Hewlett  Packard  personnel.  When  the  designers 
from  research  and  development  have  an  idea  for  a  new 
product,  a  proposal  is  sent  to  management.  If  permission  is 
given,  a  product  design  group  is  formed  consisting  of  people 
from  marketing,  research  and  developaent,  manufacturing,  and 
quality  assurance.  When  the  design  Ls  laid  out,  the  quality 
assurance  people  ask  "Waat  if"  questions  to  ensure  all 
aspects  are  considered.  The  company  sets  no  particular 
specifications  to  which  the  designers  must  adhere,  so  they 
have  the  freedom  to  be  creative.  The  main  languages  used  by 
the  designers  are  assembler,  Pascal,  and  F3RTEAN  because 
their  products  tend  to  be  more  technical  than  commercial  in 
nature.  They  also  produce  environmental  and  applications 
software.  One  person  from  guality  assurance  is  assigned  to 
each  project. 

During  the  requirement  phase  of  the  development 
process,  an  investigation  has  to  be  completed  ir.  order  to 
produce  a  detailed  specification  plan  and  a  user  interface 
specification  plan.  In  the  external  design  segment  the 
quality  assurance  people  must  produce  a  quality  plan  deli¬ 
neating  the  guality  goals  or  objects  of  the  project  and  how 
they  are  to  be  measured.  This  is  a  problem  area  for  the 
quality  assurance  people  because  if  the  product  is  generated 
at  a  customer's  request,  the  request  is  usually  not  specific 
or  incomplete.  It  is  important  that  formalized  communica¬ 
tions  be  established  to  eliminate  this  problem. 

In  the  internal  design  phase  of  the  development 
cycle,  the  internal  specifications,  top  down  design,  and 
submodule  design  take  place.  The  quality  assurance 
personnel  set  up,  monitor,  and  participate  in  design  reviews 
and  code  reviews  held  during  This  period.  They  also  produce 
the  functional  test  plan. 
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Pigure  4.4  Hewlett  Pi-fcari  Software  Development  Cycle 

Source:  Interview  u  Hewlett  Packard  Software  Qua  lit'-  Assurance 
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Daring  the  implema ntation  statement,  the  quality 
assurance  people  set  up  the  systems  test  plan.  Actual 
testing  is  not  accomplished  until  the  integration  and  *est 
segments,  and  it  is  done  on  the  function  and  systems  levels. 
Although  the  functional  test  plan  is  produced  by  the  quality 
assurance  people,  the  actual  testing  is  done  by  the 
designers  themselves.  This  level  of  testing  is  viewed 
mainly  as  a  debugging  exercise  and  would  be  a  waste  of  the 
quality  assurance  organization^  time  and  resources  if  done 
by  them.  At  the  systems  level  of  fasting,  the  plan  and 
tests  are  done  by  the  quality  assurance  people.  These  tests 
are  viewed  as  a  third  party  auditing  inspection  of  the 
product.  This  third  party  testing  is  done  because  Hewlett 
Packard  does  not  believe  that  the  program  designers  and 
analysts  can  be  completely  objective  about  their  product. 
The  qualify  assurance  group  is  also  responsible  for  the 
packaging  of  all  test  plans  for  reusability.  There  are  no 
percentages  of  correctness  sought  during  these  testing 
levels.  When  this  segment  is  complete,  the  product  is 
considered  100^  correct.. 

According  to  the  quality  assurance  people,  another 
problem  area  is  the  schedule  planning.  The  designers  do  not 
think  that  problems  will  occur  during  this  testing  phase,  so 
they  have  to  be  careful  to  plan  for  extra  time  if  problems 
occur. 

After  the  quality  certification  segment,  which  is 
basically  a  customer  acceptance  inspection,  and  the  produc¬ 
tion  certification  segment,  comes  the  manufacturing  segment. 
During  this  segment  a  pilot  run  is  made  on  the  product  to 
ensure  that,  if  a  customer  requested  the  product,  all  the 
materials  —  the  product  itself,  user  manuals  and  any  other 
items  --  are  shipped. 
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3 .  OEsiatioas 

Hewlett  Packard  believes  in  "cradle  to  grave" 
involvement  with  its  software  products,  which  aeans  they  do 
not  abandon  their  customers  after  sale.  All  Hewlett  Packard 
software  is  copyrighted  so  if  there  are  any  problems  after 
it  is  in  operation,  the  cost  to  the  customer  is  $100  per 
hour  for  repairs  unless  the  customer  has  a  subscription 
service.  Subscription  service  entitles  the  customer  to  have 
software  repaired,  updated,  or  replaced  at  a  lower  f«e. 
This  service  includes  a  plan  by  which,  if  a  program  is 
updated  or  fixed  for  any  customer,  the  updated  version  is 
sent  out  to  all  other  customers  who  have  the  same  program. 
The  decision  to  use  it  within  the  customer's  system  is  left 
to  the  customer. 

If  there  is  a  problem,  the  customer  first  notifies 
the  field  activity  which,  if  necessary,  creates  a  "work 
around"  program  to  keep  the  customer's  system  operational. 
From  the  field  activity,  the  problem  is  referred  tc  the 
manufacturer,  via  support,  and  eventually  to  the  people  in 
research  and  development  who  design  the  program.  They 
prioritize  the  problem  and  place  it  in  their  schedule,  and 
it  is  eventually  fixed.  Wo  historical  records  of  problems 
or  changes  to  programs  are  maintained. 

The  quality  assurance  organization  keeps  abreast  of 
the  latest  ideas  and  changes  in  this  field  and  is  constantly 
striving  to  improve  its  prograi. 

SsSsiSUSe 

Personal  interview  with  Hr.  Rayiond  L.  Spear,  software 
produces  assurance  manager,  at  the  Hewlett  Packard  plant. 
Data  Systems  Division,  Cupertino,  California,  on  14  April 
199  2. 
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B.  TRW 

TRW  is  a  diversified  « ultinational  company  which  manu¬ 
factures  a  wide  range  of  products  from  components  for  cars 
and  trucks  to  defense  electronics  and  space  systems.  TRW 
produces  transistors,  resistors,  capacitors,  diodes,  poten¬ 
tiometers,  trimmers,  tuning  devices,  TV  convergence  yokes, 
connectors,  transformers,  printed  circuit  beards,  electric 
motors,  electric  data  processing  terminals,  and  jet  engine 
parts.  Other  products  include  pumps,  fluid  handling  equip¬ 


ment,  nuclear  reactor  components,  fastners,  bearings, 
cutting  tools,  and  hand  tools. 

This  company  handles  defense  systems  contracts  which 
include  the  development  of  software  and  the  construction  of 
tha  entire  system. 

1 .  Organization 

TRW  is  divided  into  many  groups  because  of  its 
diversification.  One  of  these  groups,  the  defense  systems 
group,  contains  the  engineering  division  of  which  the  prod¬ 
ucts  assurance  organization  is  a  part.  (Figure  4.5)  This 
level  is  made  up  of  managers  who  are  assigned  to  the 
different  projects  in  assistant  project  manager  capacity. 
This  department  is  not  just  concerned  with  software  product 
assurance,  but  also  with  hardware  and  system  engineering  and 
design  (SEAD)  product  assurance.  (Pigure  4.6) 

Figure  4.7  shows  the  standard  work  breakdown  struc¬ 
ture  for  any  product  in  the  defense  systems  group  as  it  is 
concerned  with  product  assurance.  The  assistant  project 
manager  heads  up  a  staff  of  personnel  who  work  ir.  the  areas 
of  quality  assurance,  configuration  nar.agement,  reliability, 
and  safety. 

Figure  4.8  shows  the  standard  work  breakdown  struc¬ 
ture  for  the  quality  assurance  area  of  the  project  which  is 
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FIGURE  4.5  TRW  Defense  Systems  Group 

SOURCE:  Interview  with  TRW  Products  Assurance  Personnel 


FIGURE  4.6  TRW  Products  Assurance  Department 
SOURCE:  Interview  with  TRW  Products  Assurance  Manager 


FIGURE  4.8  TRW  Quality  Assurance  Standardized  Project  WBS 

SOURCE:  TRW  Status  Report  on  Standardization  of  Quality  Assurance  Functions  Task, 
20  April  1982 


further  subdivided  into  management,  software,  hardware  and 
system. 

When  working  on  military  contracts,  the  company  must 
follow  specifications  required  for  contract  award.  One  of 
these  specifications  is  MIL-S-52779A  "Software  Quality 
Assurance  Program  Requirements"  of  1  August  1979.  This 
document  states  the  requirement  for  the  establishment  and 
implementation  of  a  software  quality  assurance  program.  It 
is  hoped  that  this  program  could  be  tailored,  economically 
planned  and  developed  in  conjunction  with  the  contractors 
programs  of  this  type.  The  contractor  is  required  to  docu¬ 
ment  this  program  in  the  form  of  a  software  quality  assur¬ 
ance  plan  which  meets  its  specifications.  This  plan  has  to 
identify  organizational  rssponsibilit y  and  authority  for  its 
execution  and  make  timely  provisions  for  special  needs 
(controls,  tools,  facilities,  skills,  etc.).  Because  this 
is  part  of  the  contract,  i t  is  considered  to  give  the  prod¬ 
ucts  assurance  organization  its  authority  over  the  project. 

2.  Management  and  Software  Areas  of  the  Project 

The  standard  duties  expected  to  be  performed  by  the 
personnel  in  the  management  area  of  the  project  are  as 
follows:  (Figure  4.9) 

a.  Planning  and  Control 

(1)  To  provide  direction  and  participate  in  the 
generation  of  quality  assurance  input  into  the  project 
implementation  plan,  project  schedules,  documentation  plans 
and  other  similar  documents. 

(2)  To  define  the  quality  assurance  tasks  and 
assign  the  appropriate  personnel.  To  monitor  their  perform¬ 
ance  and  prepare  status  reports. 
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FIGURE 

SOURCE: 


•  SELECTION 

•  REQUIREMENTS  DEFINITION 

•  MONITORING  AND  CONTROL 


.9  TRW  Quality  Assurance  Project  WBS  -  Management  Detail 

TRW  Status  Report  on  Standardization  of  Quality  Assurance 
Functions  Task,  20  April  1982 
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(3)  To  aonitor  all  actions  in  conjunction  with 
contract  and  engineering  changes. 

b.  Quality  Assurance  Plans  and  Procedures 

They  are  required  to  direct  the  generation  of 
the  quality  assurance  pLan  which  follows  the  controlling 
government  specification  SI L-S-52779A,to  review,  maintain, 
and  update  it  throughout  the  project's  life.  This  plan  is 
required  to  address: 

(1)  Tools,  techniques,  methodologies  and 
records  to  be  employed  in  the  performance  of  the  work  to 
support  the  quality  assurance  objectives. 

(2)  Procedures  by  which  design  documentation  is 
reviewed  to  evaluate  design  logic,  fulfillment  of  require¬ 
ments,  completeness,  and  compliance  with  specified 
standards. 

(3)  Contractor's  procedures  for  formally 
approving  or  certifying  the  description,  authorization  and 
completion  of  work  performed  under  contract. 

(4)  Documentation  of  standards,  programming 
conventions  and  practices  to  be  used  for  all  software. 

(5)  Documentation  of  the  contractor's  proce¬ 
dures  and  controls  for  handling  of  source  cods  and  object 
code  and  related  data  in  their  various  forms  and  versions. 

(6)  Documentation  of  contractor's  procedures 
for  preparation  and  execution  of  reviews  and  audits  neces¬ 
sary  in  establishing  traceability  of  initial  contract 
requirements. 

c.  Project  Interfaces 

The  management  detail  addresses  the  interfaces 
between  project  manager,  assistant  project  manager,  sub 
project  managers  and  others  in  conjunction  with  the  project. 
They  attend  the  staff  meetings  and  respond  to  action  items. 
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d.  Customer  Interfaces 


The  management  detail  works  with  the  customer 
representative  offices,  hosts  their  visits  and  formal 
reviews  and  take  care  of  documentation  to  and  from  the 
customer. 


e.  Subcontractor  and  supplier  Management 

Figure  4.10  delinates  the  duties  of  the 
personnel  in  the  software  area  of  the  project.  The  three 
groupings  are: 

(1)  Management  Support  --  carries  out  duties 

in  support  of  the  management  section  of 

the  project. 

(2)  Engineering 

(a)  Identify  and  define  *he  quality- 
standards  and  procedures  that  will 
be  followed  during  the  design, 
development,  programming,  testing 
and  documentation  stages. 

(b)  Identify  software  tools  and  special 
methodologies  that  would  be  used  in 
performance  of  quality  assurance 
task.  Establish  procedures  for  their 
use  and  ensure  their  use  during  the 
proje  ct. 

(c)  Participate  in  definition  and  implemen¬ 
tation  of  a  software  problem  reporting, 
analysis,  correction  and  control  system. 

(d)  Participate  in  formal  reviews,  project 
boards  and  customer  boards. 

(e)  Maintain  records  and  files  of  documen¬ 
tation  review  for  adherence  to 
standards. 

(3)  Operations 
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SOFTWARE 


MANAGEMENT 

SUPPORT 


•  PROJECT  PLANNINC  AND  CONTROL 

•  QA  PLAN  AND  PROCEDURES 

•  PROJECT  INTERFACES 

•  CUSTOMER  INTERFACES 

•  SUBCONTRACTOR /SUPPLIER  INTERFACES 


ENGINEERING 


•  S/W  STANDARDS  AND  PROCEDURES 

•  S/W  TOOLS  AND  METHODOLOGIES 

•  S/W  PROBLEM  CONTROL 

•  FORMAL  REVIEWS 

•  PROJECT  BOARDS 

•  RECORDS  MAINTENANCE 

•  DOCUMENTATION  REVIEW 


OPERATIONS 


•  AUDITS 

•  TESTS 

•  INSPECTIONS 

•  SITE  SUPPORT 


FIGURE  4.10  TRW  Quality  Assurance  Project  WBS  -  Software  Detail 

SOURCE:  TRW  Status  Report  on  Standardization  of  Quality  Assurance 
Functions  Task,  20  April  1982 
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(a)  Perform  audits  on  project  activities. 

(b)  Participate  in  each  level  of  software 
tasting  as  dasigned  by  tha  quality 
assurance  plan  and  perform  survaillanca 
activities. 

(c)  Perform  visual  inspections  of  all 
software  products  purchased  with  hard¬ 
ware  from  supplier. 

(d)  Perform  quality  assurance  function  at 
each  site  and  :emo*e  site  for  testing. 

If,  during  any  documentation  audit,  a  discrepancy  is  found, 
tha  discrepancy  is  documented  and  is  taken  first  to  the 
responsible  designer.  If,  in  a  certain  amount  of  time,  the 
error  is  not  corrected,  the  problem  is  taken  to  the  next 
level  in  the  project  organization.  The  problem  will  travel 
up  the  organization  until  the  discrepancy  is  corrected  even 
if  it  means  going  outside  the  project's  environment. 

Approximately  2  to  5.5 %  of  the  entire  project's 
funds  is  charged  to  quality  product  assurance,  but  it  is  the 
opinion  of  the  managers  of  quality  assurance  in  the  TRW 
company  that  the  cost  of  quality  assurance  is  zero. 

Once  a  product  has  been  accepted  by  the 
customer,  with  the  signing  of  defense  form  DD250  Material 
Inspection  and  Receiving  Report,  the  legal  obligation  of  TRW 
is  ended.  If  any  problems  arise  after  release,  the  customer 
pays  to  have  more  work  done. 

Re£22£££e 

Personal  interview  with  Mr.  William  7.  Buck,  Product 
Assurance  Manager;  Mr.  Samuel  E.  Benesch,  Department  Manaaer 
Product  Assurance;  and  Mr.  Martin  F.  Kenehan,  Senior  Staff 
Engineer  of  the  Defense  and  space  Sroup  of  TRW,  Redondo 
Beach,  California  on  7  May  1982. 
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C.  IBM 


1 .  Organ iza.t  ion 

Figure  4.11  shows  the  structure  of  the  IBM  organiza¬ 
tion  as  of  March  1982.  I*  shows  that,  under  the  staff 
level,  the  company  is  divided  Into  two  lajor  areas, 
marketing  and  service  and  manufacturing  and  development. 
Under  these  areas,  the  grouping  of  divisions  start  in  which, 
under  the  information  systems  and  technology  group,  the 
general  products  division  exists. 

The  general  products  divisioa,  with  its  headquarters 
located  in  San  Jose,  California,  is  responsible  for  the 
development  of  all  hardware  and  software  products  at  IBM. 
It  has  two  development  laboratories,  one  located  in  Santa 
Teresa,  California  and  the  other  in  Tucson,  Arizona.  (Figure 
4.12) 

The  general  products  division  is  headed  by  a  presi¬ 
dent  with  a  vice-president  in  charge  of  each  operational 
department  including:  hardware,  software,  manufacturing, 
financing,  support  and  products  assurance.  Heading  each 
development  laboratory  is  a  center  aanager  with  functional 
managers  in  charge  of  each  department  below  him.  within 
each  of  the  development  centers,  a  functional  manager  in 
charge  of  products  or  quality  assurance. 

The  quality  assurance  department  within  this  organi¬ 
zation  is  completely  independent  of  other  departments.  The 
software  products  developed  in  these  laboratories  lie  within 
the  environment  or  operational  tool  area  (Figure  4.13)  and 
they  are  produced  in  all  of  the  major  programming  languages. 
The  quality  assurance  group  does  have  authority  over  prod¬ 
ucts  that  are  new  and  are  about  to  be  announced  and  over 
products  that  are  being  shipped  to  customers.  If  this  group 
does  not  agree  that  a  product  is  ready,  it  is  not  released. 


72 


Tha  decision  for  product  calease  is  not  driven  by  any  other 
factors. 

The  quality  assurance  department  is  divided  into 
three  divisions,  two  of  which  are  products  assurance,  and 
tha  other  is  verification  and  testing.  Every  software 
product  developed  is  divided  between  the  two  product  assur¬ 
ance  divisions.  The  number  of  people  assigned  is  a  function 
of  the  project's  size  and  their  schedule  depends  or.  that  of 
the  developers.  At  the  end  of  the  development  cycle,  all 
products  go  through  the  verification  and  testing  division. 

2 .  Quality  Assurance  and  Design  Interface 

The  quality  assurance  group  interfaces  with  the 
program  developers  throughout  the  entire  development  cycle. 
(Figure  4.14)  The  people  within  tais  group  have  no  prere¬ 
quisite  shill  requirement  and  most  lave  varied  backgrounds 
ranging  from  programming  expertise  to  marketing  skills.  To 
do  their  job,  they  depend  mainly  on  their  experience  and  gut 
feelings.  It  is  not  considered  necessary  for  them  to  have  a 
programming  or  computer  engineering  background  because  it  is 
very  rare  that  they  have  to  inspect  the  actual  code  itself. 
Within  each  development  department  are  performance  groups 
who  examine  the  code  and  test  it  periodically  throughout  the 
development  cycle. 

The  managers  of  the  development  groups  depend  on  the 
people  from  products  assurance  for  their  objectivity  and  do 
not  view  them  as  a  resource  tool.  These  products  assurance 
people  contribute  to  the  product  in  the  following  ways: 

a.  Planning 

Before  any  work  can  be  started,  a  project  plan 
has  to  be  put  together  in  which  the  programmers  have  to 
claim  which  development  style  out  of  a  possible  three  will 
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FIGURE  4.14  IBM  Software  Development  Process 

SOURCE:  Interview  with  IBM  Products  Assurance  Personnel 


be  used  for  this  project.  This  plan  is  naaed  the 
Comprehensive  Evaluation  Plan  (CEP|  which  also  takes  into 
account  the  quality  assurance  procedures,  use  of  resources, 
ar.d  the  project's  schedule.  It  is  considered  the  main  plan¬ 
ning  document  and  has  to  be  approved  by  the  products  assur¬ 
ance  division  before  the  project  is  started. 

b.  Early  Warnings 

If  at  any  time  during  the  development  cycle,  the 
guality  assurance  inspector  sees  anything  which  might  keep 
the  program  development  group  from  keeping  schedule,  they 
notify  the  project  manager. 

c.  Value  Added 

If,  during  the  process,  the  quality  assurance 
people  feel  that  something  could  be  added  to  the  software  to 
enhance  or  improve  it,  they  inform  the  development  group. 

d.  Education 

The  education  of  the  programmers  on  possible 
development  tools,  whether  developed  in  house  or  externally, 
is  carried  out  by  this  organization. 

IBM  sets  standards  requirements  that  have  to  be 
built  into  the  products,  but  there  is  flexibility  in  their 
use  because  it  is  left  to  the  discretion  of  the  programmer. 

The  verification  and  testing  people  carry  out 
their  functional  testing  at  the  end  of  the  development 
process,  performing  basically  user  oriented  tests.  Their 
main  objective  is  to  debug  these  products  of  any  user 
oriented  problems. 

Besides  the  product  assurance,  performance 
group,  and  verification  and  test  groups  interfaces,  there  is 
still  another  built-ir.  device  for  insuring  quality  products. 


A  Review  a nd  Inspection  process  (RSI)  is  carried  cut  by  the 
programmers  themselves  throughout  the  development  cycle.  It 
is  carried  out  either  in  a  foraal  aanner  in  which  a  nesting 
is  held  with  the  prograasers  and  a  sod  rator  and  they 
discuss  the  prograa  and  its  progress  in  depth,  or  it  can  be 
held  on  an  inforaal  basis  with  only  the  prograasers'  imae- 
diate  peers  present,  k  representative  of  the  product  assur¬ 
ance  division  is  required  to  attend  these  nestings. 

3.  Operations 

Cnee  a  product  has  been  released,  the  field  engi¬ 
neering  division  is  responsible  for  remedying  any  problems 
experienced  by  the  customers  in  use  of  the  product.  This 
division  is  also  responsible  for  maintaining  a  historical 
tracking  record  on  problens  with  the  software  products  once 
in  the  field.  If  a  product  is  to  be  renewed  or  enhanced, 
the  products  assurance  people  can  request  this  historical 
information,  but  they  are  not  required  to  keep  track  of  it. 

If  a  completely  new  product  is  released  by  the 
company  for  which  the  users  would  require  training,  the 
responsibility  for  this  training  is  assumed  by  the  marketing 
division.  Requests  for  new  products  are  not  received 
directly  by  the  dev«lopmen t  laboratories,  but  through  the 
two  main  IBS  user  groups,  SHARE  and  30IDE,  which  meet  twice 
yearly  to  discuss  problems  and  possible  ideas  for  new  prod¬ 
ucts.  The  marketing  division  is  also  constantly  carrying 
out  surveys  of  customers  for  new  product  ideas. 

The  people  of  the  quality  assurance  department 
thought  that  their  main  objective  was  tc  maintain  a  wide 
range  perspective  of  the  product  development  process  and 
never  to  become  overly  involved  with  details. 


McDonald  and  Mr. 
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Personal  interview  with  Mr.  Barron  A. 

Norman  Towns  of  the  products  assurance  group,  IBM 
Development  Center,  Santa  Teresa,  California  on  21  April 
1982. 

D.  AMDAHL 

Amdahl  is  a  high  technology  company  engaged  in  the 
stare-of-t he-art  design,  development,  manufacturing, 
marketing  and  maintenance  of  large  mainframe  computers, 
software  and  communication  systems.  These  products  are  used 
by  large  computer  users  in  the  full  -pectrum  of  commercial 
and  scientific  data  processing  environments. 

The  company's  central  processing  unit's  design  strategy 
is  to  focus  on  the  development  of  efficient  design  architec¬ 
ture  for  high  performance,  dependability,  and  flexibility 
for  future  enhancement  of  the  product. 

The  company's  communication  systems  division  designs  and 
manufactures  digital  communication  networks  which  allow 
users  to  interface  with  multiple  geographically  dispersed 
systems. 

Amdahl  also  offers  a  number  of  services  to  its 
customers.  There  are  programs  for  cross  training  support 
with  specialists  in  both  hardware  and  software  disciplines. 
There  are  also  expanded  educational  offerings  with  tailored 
traininq  to  enhance  Amdahl  product  support. 

The  company's  software  development  and  program  enhance¬ 
ments  ensure  compatibility  of  its  hardware  products  tc  the 
most  widely  used  systems,  and  other  software  products  are 
aimed  at  increasing  productivity  of  the  user. 
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The  software  department  is  a  part  of  the  engineering 
division  at  Amdahl.  The  software  quality  assurance  group  is 
a  part  of  that  department  and  it  consists  of  five  people. 
(Figure  4.15)  The  main  purpose  of  software  in  the  Amdahl 
world  is  for  architectural  interface  of  its  product  with  the 
customer’s  system.  Because  of  this,  the  software  develop¬ 
ment  group  does  not  have  to  start  with  any  top  down  design 
of  its  product  but  to  develop  complement  software  in  order 
to  tie  the  hardware  products  together.  The  driving  force 
for  the  development  of  software  in  this  company  is  the  inno¬ 
vative  hardware  of  its  competitors,  such  as  IBM.  The 
authority  of  this  organize tion  depends  on  its  credibility 
and  expertise.  The  products  that  they  release  have  proven 
themselves  in  the  market  place. 

2.  Development  Interface 

The  quality  assurance  group  of  Amdahl's  main  inter¬ 
face  with  the  development  group  cones  at  the  end  of  the 
development  cycle  during  the  testing  and  measuring.  They 
also  take  part  in  all  technical  reviews  throughout  c he  new 
products  development.  The  quality  assurance  group  insures 
■chat  the  program  is  "packaged  correctly"  for  inscallation. 
This  means  that  the  software  product  meets  all  the  standards 
of  their  competitor's  system. 

3 .  Operations 

For  new  software  about  to  be  released  by  this 
company,  they  have  what  is  known  as  the  early  support 
program.  The  program  enables  the  developers  to  take  the 
software  into  the  field,  test  and  debug  it  on  the  system  to 
which  it  is  to  be  applied  before  it  is  announced. 
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FIGURE  4.15  Amdahl  Software  Department 

SOURCE:  Interview  with  Amdahl  Software  Quality  Assurance  Manager 


After  installation,  if  there  are  problems  with  the 
software,  the  field  units  of  Amdahl  handle  them.  There  is 
the  Amdahl  warning  system  and  maintenance  tape,  which  is 
maintained  by  the  field  units  and,  if  there  is  a  major 
problem,  the  software  is  sent  back  to  the  development  center 
for  rework. 

No  training  is  carried  out  for  the  Amdahl  products, 
but  there  is  a  tremendous  in-house  training  effort  on 
competitors*  equipment. 

Reference 

Interview  with  Sr.  Richard  L.  Patrick,  Manager,  Software 
Quality  Assurance  Group  at  Amdahl's. 
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V.  A  NA  LYSIS  CONCLUSION  AND  RECOMMENDATIONS 

This  chapter  gives  the  reader  an  analysis  and  summary  of 
the  interviews  with  the  commercial  computer  companies  and  a 
comparison  with  the  FMSO  environment.  At  the  end  of  the 
chapter,  conclusions  and  recommendations  are  given. 

A.  QUESTIONS  FHOM  INTERVIEW: 

1.  Where  does  the  quality  assurance  group  fit  into  the 
company's  organization? 

a.  Hewlett  Packard 

The  products  assurance  group  is  a  part  of  the 
data  systems  division  and  is  on  the  same  level  as  engi¬ 
neering,  manufacturing,  marketing,  and  other  departments  of 
this  division.  The  products  assurance  group  fits  into  the 
company's  organization  in  a  line  function  position. 

b.  TRW 

The  products  assurance  group  is  a  part  of  the 
engineering  division.  This  group  fits  into  the  company's 
organization  in  a  staff  function. 

c.  IBM 

The  products  assurance  department  is  a  part  of 
the  software  development  center.  It  is  positioned  on  the 
same  level  as  the  development  department  cf  the  center,  in  a 
line  function. 

d.  Amdahl 

The  software  quality  assurance  group  is  a  part 
of  the  software  department .  It  is  positioned  on  the  same 
level  as  the  research  and  development  groups.  The  software 
quality  assurance  group  is  in  a  line  function  position. 
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e.  FMSO 

The  quality  control  branch  exists  in  the  manage¬ 
ment  department,  Code  92.  It  is  in  a  staff  function. 

2.  What  type  of  author ity/power  does  the  quality  assur¬ 
ance  group  have  over  their  software  product? 

a.  Hewlett  Packard 

This  group's  power  relies  on  its  ability  to 
persuade  management  that  the  product  is  not  ready  and  its 
reputation. 

b.  TRW 

The  authority  of  this  quality  assurance  group  is 
given  by  a  contractual  requirement,  MIL-S-52779 A  "Software 
Qualify  Assurance  Program  Requirements." 

c.  IBM 

The  products  assurance  group  has  complete 
authority  over  software  product.  If  this  group  feels  that 
the  product  is  not  ready,  it  is  not  released. 

d.  Amdahl 

The  software  quality  assurance  group's  power 
over  the  product  depends  on  the  group's  credibility  and 
exp  ertise. 

e.  FMSO 

The  quality  control  group  exercises  administra¬ 
tive  power  over  products.  It  insures  that  the  quality 
assurance  check-off  list  is  properly  filled  out  and  that  the 
product  meets  specifications. 

3.  What  qualifications  do  the  people  in  the  quality 
assurance  group  have? 

a.  Hewlett  Packard 

Their  quality  assurance  personnel  are  required 
to  have  enough  education  and  experience  to  be  programmers 
and  designers. 

b.  TRW 

No  specific  qualification  required. 
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C.  IBM 

No  specific  qualification  required. 

d.  Amdahl 

No  specific  qualification  required. 

e.  FMSO 

Personnel  in  the  quality  control  branch  are 
expected  to  have  a  complete  knowledge  of  the  system  develop¬ 
ment  process,  from  all  aspects. 

4.  How  does  the  quality  assurance  group  interface  with 
the  design/developaent  group? 

a.  Hewlett  Packard 

The  quality  assurance  personnel  are  a  part  of 
the  product  development  group  and  work  with  the  product 
designers  throughout  the  development  cycle.  They  are 
required  to  produce  a  qualify  assurance  plan  which  states 
the  measurements  of  the  quality  objectives  and  to  partici¬ 
pate  in  the  product  testing  or.  both  the  functional  and 
system  levels. 

b.  TRW 

An  assistant  project  manager  is  assigned  to 
every  project,  with  his  own  staff,  to  coordinate  and  partic¬ 
ipate  ir.  the  quality  control  functions  required  in  the 
project.  They  perform  audit  testing  of  the  product  and 
participate  in  all  technical  reviews. 

C.  IBM 

The  product  assurance  people  interface  with  the 
software  development  personnel  throughout  the  development 
cycle.  They  approve  the  program  development  plan  and  keep 
management  informed  of  anything  that  might  affect  the 
project's  schedule.  They  do  not  participate  in  product 
tesfing,  but  there  are  two  third  party  groups,  the  perform¬ 
ance  group  and  the  verification  ar.i  test  personnel,  who 
carry  out  this  function. 
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d.  Amdahl 

The  software  quality  asarance  group  interfaces 
with  the  development  personnel  at  the  testing  and  measure¬ 
ment  end  of  the  development  cycle.  They  insure  that  the 
product  is  "packaged  correctly"  before  release.  They  are 
required  to  attend  and  participate  in  all  technical  reviews 
during  the  development  of  the  product. 

e.  FMSO 

The  quality  control  branch  checks  the  functional 
description  and  system  specificat ions  administratively. 
They  insure  that  the  quality  control  check-off  list  is 
filled  out  properly  and  participate  in  product  testing  on  a 
very  infrequent  basis. 

As  shown  in  the  question,  all  of  those  interviewed, 
except  TRW  and  FMSO,  had  their  software  quality  assurance 
groups  in  a  line  function  position  in  the  organization.  It 
should  be  noted  that  the  products  assurance  group  of  TRW  was 
in  charge  of  a  line  management  staff  which  was  assigned  to 
each  product  to  perform  in  a  line  function.  In  FMSO,  there 
is  only  the  staff  group. 

It  is  "he  opinion  of  the  author  of  this  thesis  that 
questions  2,  3,  and  4  tie  in  together.  In  all  the  companies 
interviewed,  the  quality  assurance  group  is  considered  and 
functions  as  an  integral  part  of  the  development  team.  They 
work  with  the  development  personnel  throughout  the  develop¬ 
ment  cycle,  relieving  any  advisary  situation. 

If  the  personnel  in  the  quality  assurance  group  do  not 
have  the  expertise  to  carry  out  testing  of  the  product,  a 
third  party  in  the  company's  organization  do.  Development 
personnel  cannot  be  expected  to  be  completely  objective 
about  their  own  product  no  perform  its  testing. 

Because  the  quality  assurance  personnel  work  alongside 
the  development  people  and  perform  some  form  of  audit 
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function,  their  opinion  has  credibility  with  the  developmenf 
people  and  management.  This  has  a  direct  effect  on  their 
authority  over  the  product. 

In  FMSO,  the  quality  control  branch  does  sot  become  an 
integral  part  of  the  development  team.  They  rarely  perform 
any  auditing  function  on  the  product.  The  development 
people  in  the  CDAs  carry  out  all  testing.  If  the  quality 
assurance  check-off  list  is  completely  filled  out,  the 
quality  control  branch  has  no  real  justification  for  stop¬ 
ping  the  product's  release. 

5.  What  fools,  methodologies,  or  techniques  does  the 
quality  assurance  group  use  to  do  their  job? 

a.  Hewlett  Packard 

No  tools,  methodologies  or  techniques  were  used 
tha*  were  unique  to  the  quality  assurance  function. 

b.  TRW 

No  tools,  methodologies  or  techniques  were  used 
that  were  unique  to  the  quality  assurance  function. 

c.  IBM 

No  tools,  methodologies  or  techniques  were  used 
that  were  unique  to  the  qulity  assurance  function. 

d.  Amdahl 

No  tools,  methodologies  or  techniques  were  used 
that  were  unique  to  the  quality  assurance  function. 

e.  FMSO 


No  tools,  methodologies  or  techniques  were  used 
that  were  unique  to  the  quality  assurance  function. 

On  this  question,  none  of  the  companies  iatervived 
stated  that  they  used  anything  unique  to  the  quality  assur¬ 
ance  function.  The  quality  assurance  personnel  were 
knowledaable  of  fools  and  techniques  that  could  be  used  by 
the  development  programmers  which,  from  their  viewpoint, 
aided  in  the  quality  of  the  software  because  it  heloed  the 


programmers  write  better  programs.  These  tools  and  techni¬ 
ques  were  acquired  through  the  survey  of  computer  science 
literature  or  developed  within  the  company  and  passed  on. 
No  company  interviewed  was  willing  to  share  any  of  these 
tools  with  the  author  of  this  thesis  because  their  tools 
were  of  a  proprietary  nature. 

There  are  companies  that  develop  tools  and  provide 
services  which  aid  in  the  areas  of  programming  and  quality 
assurance.  One  such  company  is  Software  Research  Associates 
(SR A) ,  headquartered  in  San  Francisco,  California.  A 
description  of  the  purpose  of  this  company  and  its  activi¬ 
ties  is  provided  in  Appendix  B. 

6,  Are  historical  records  kept  of  problems  with  soft¬ 
ware  products  after  their  release  and  who  in  the  company's 
organization  keeps  them? 

a.  Hewlett  Packard 

No  records  of  this  type  are  being  kept  at  this 

time. 

b.  TRW 

No  records  are  kept  of  product  problems  after 

release. 

c.  IBM 

Historical  records  of  problems  are  kept  by  the 
field  engineering  division. 

d.  Amdahl 

A  maintenance  tape  of  problems  is  kept  by  the 
field  engineering  division. 

e.  FHSO 

Records  are  maintained  ny  the  quality  control 
branch  through  analysis  of  Program  Trouble  Reports  (PTR) . 

7.  Who  handles  problems  with  software  after  release, 
and  how  arr  such  problems  handled? 


a.  Hewlett  Packard 

Problems  are  handled  by  field  engineering  activ¬ 
ities  who  build  "work  arounds"  for  customers  if  necessary. 
If  there  is  a  critical  problem,  the  software  is  returned  to 
the  development  group  for  repair. 

b.  TRW 

There  is  no  legal  obligation  on  the  part  of  the 
company  to  handle  problems  after  a  product's  release.  If  a 
customer  desires  TRW  to  fix  a  problea  after  product  release, 
the  customer  will  be  charged  for  the  services. 

c.  IBM 

All  problems  are  completely  handled  by  t.-*e  field 
engineering  division.  The  software  is  not  returned  tc  the 
development  laboratory,  no  matter  how  critical. 

d.  Amdahl 

Problems  are  handled  by  the  field  engineering 
group.  If  there  is  a  aajor  problem,  the  software  is 
returned  to  the  development  personnel. 

e.  FMSO 

The  software  is  reported  tc  the  CDA  and 

repaired. 

8.  If  a  brand  new  product  is  designed,  who  in  the 
coapany's  organization  trains  the  customer  on  this  product? 

a.  Hewlett  Packard 

Marketing  division  carries  out  training. 

b.  TRW 

Ho  training  is  carried  oit  by  the  company  after 
product  release. 

c.  IBM 

Marketing  division  carries  out  training. 

d.  Amdahl 

Marketing  division  carries  out  training. 
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e.  FMSO 

Field  training  units  go  to  activities  from  the 

CDAs. 

A  question  that  might  have  been  asked  during  these 
interviews  concerned  the  effectiveness  of  the  company's 
software  quality  assurance  program.  The  author  did  not  ask 
this  question  because  it  would  be  improbable  to  expect  an 
objective  answer.  This  thesis  did  not  offer  a  quantitative 
measure  of  these  groups'  performances  to  make  its  compari¬ 
sons.  The  author's  intent  was  to  compare  their  view  of  the 
quality  assurance  organization's  role  and  how  they  function. 

B.  SUMMARY  CONCLUSIONS 

The  purpose  of  this  thesis  was  to  investigate  the 
methods  used  by  large  commercial  computer  companies  in  the 
area  of  software  quality  assurance.  The  primary  objective 
was  to  see  if  any  of  these  practices  could  be  used  in  FMSO's 
environment. 

1.  The  greatest  difference  between  the  commercial 
companies  and  the  FMSC  environment  was  in  management's  view 
of  what  role  or  function  a  quality  assurance  group  should 
take.  In  the  commercial  environment,  the  trend  of  thought 
is  that  the  quality  assurance  role  is  a  line  function  that 
could  be  controlled  from  a  staff  position.  In  FMSO,  the 
quality  assurance  role  is  only  being  fulfilled  through  a 
staff  position. 

2.  There  was  a  difference  in  the  way  *he  quality  assur¬ 
ance  personnel  interfaced  with  the  development  people.  In 
the  commercial  companies,  the  quality  assurance  personnel 
became  an  integral  part  of  the  development  team,  their  opin¬ 
ions  and  actions  being  a  very  valuable  management  device  to 
project  managers.  In  FMSO,  the  quaLity  controL  branch  from 
its  staff  position,  does  not  become  a  part  of  the  develop¬ 
ment  team,  thus  creating  an  adversary  environment. 
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C.  RECOMMENDATIONS 


1.  It  is  the  opinion  of  the  author  that  FC1SO  should 
change  the  quality  control  branch's  position  from  a  staff  to 
a  line  function.  As  shown  by  the  interviews,  this  is  the 
trend  of  thought  on  the  position  of  an  organization  of  this 
type  in  a  software  production  environment  of  today. 

2.  In  the  PHSO  environaent,  to  concert  the  quality  control 
branch's  position  from  a  staff  to  a  line  function,  an 
increase  in  the  branch's  size  would  be  necessary. 

This  could  be  accomplished  in  two  ways.  Dne  way  would 
be  to  hire  more  people  to  increase  its  size.  The  other 
manner  would  be  to  take  people  already  in  the  CDAs  and 
assign  them  the  specific  job  of  quaLity  assurance.  The 
second  manner  may  be  acre  effective  because  these  people 
would  already  be  acclimated  to  the  F353  environaent  and  have 
the  knowledge  of  practices  in  their  own  CDA.  People  of 
experience  and  expertise  could  be  chosen  and,  since  already 
known  by  the  personnel  in  their  development  groups,  would 
not  be  viewed  as  outsiders.  They  would  be  able  no  either 
carry  out  or  be  in  charge  of  the  auditing  functions  in  the 
software  development  process.  FMSO  would  not  have  to  change 
its  development  process.  The  staff  function  or  position 
could  still  be  held  in  Code  92,  but  it  would  be  in  charge  of 
a  line  quality  assurance  organization  in  the  CDAs. 

3.  The  Qualty  Assurance  Checklist  could  be  used  as  the 
quality  assurance  group's  work  description  document.  They 
would  be  in  charge  of  carrying  out  the  elaaents  of  the 
checklist  in  a  third  party  auditing  function.  Because  the 
checklist  points  out  the  segments  during  the  development 
process  where  surveillence  for  quality  is  important  and  the 
list  covers  the  entire  development  process,  it  would  be  a 
very  useful  guideline. 
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Looking  at  the  first  element  of  the  checklist,  the  scope 
of  release,  a  separate  checklist  should  be  mads  up  for  each 
of  the  four  levels  of  projects  to  cut  down  on  confusion  of 
which  elements  should  be  done  for  which  project. 

The  elements  stated  in  the  checklist  are  also  very 
broad.  A  more  specific  description  of  the  tasks  that  would 
have  to  be  carried  out  by  the  quality  assurance  personnel 
should  be  promulgated.  This  description  of  tasks  would  also 
have  to  coinside  with  the  steps  of  the  system  development 
process. 

The  quality  assurance  staff  function  ir.  Code  9  2  should 
monitor  the  projects  progress  and  be  irvolved  in  it's  POASM 
phase.  They  should  have  final  authority  over  the  this  mile¬ 
stone  plan.  They  should  attend  all  project  internal  reviews 
and  partipate  in,  if  no  more  than  monitor,  all  testing. 

4.  With  the  quality  control  branch  in  its  present  position, 
it  is  the  opinion  of  the  author  that  it  is  a  waste  of  this 
organization's  time  and  resources  to  be  involved  in  the 
collection  and  analysis  of  Program  Trouble  Reports  which 
record  problems  after  software  release.  The  only 
organization  to  which  this  type  of  information  is  important 
is  the  organization  which  developed  it  and  has  to  fix  it. 
This  organization  should  expend  its  energy  in  the 
maintenance  of  these  types  of  records,  and  the  quality 
assurance  people  should  monitor  them. 

5.  An  effort  should  be  made  by  FSSD  to  maintain  records  of 
in-house  development  tools  that  could  be  shared  between  the 
CDAs.  The  assistance  of  a  tool  development  organization, 
such  as  Software  Research  Associates,  could  be  sought  to 
help  them  in  the  areas  of  program  development  and  software 
quality  control  tools. 

6.  If  any  justification  need  be  supplied  for  acquiring 
resources  to  accomplish  these  goals,  the  requirements 


92 


invoked  on  civilian  contractors  for  a  software  quality 
assurance  program,  MIL-S-5  2779A,  could  be  given.  If  the 
government  requires  this  extensive  a  program  for  its 
civilian  contractors,  why  not  require  it  for  itself? 


APPENDIX  A 

FMSO  SYSTEM  DEVELOPMENT  PROCESS 


2.3.2  System  Development  Process  (SDP)  is  the  function  by  which  FMSO  trans¬ 
forms  a  Requirements  Statement  into  a  documented,  functioning  set  of  computer 
programs  and  procedures.  FMSOINTNOTE  5230  of  21  Mov  1979  established  the  CDA 
Development  Process  Model  provided  as  Figure  2-9 .  The  CDA  Development  Process 
Model  reflects  all  of  the  basic  steps  appropriate  in  ensuring  that  each  CDA 
Tasking  received  by  FMSO  is  effectively  managed  and  results  in  a  high  quality 
product  being  released  for  use  by  the  customer.  The  model  covers  all  projects, 
large  and  small,  new  development  or  maintenance.  However,  it  is  anticipated 
that  some  of  the  steps  in  the  model  may  not  be  applicable  to  all  projects. 
Therefore,  an  explicit  decision  by  tae  appropriate  level  of  management  is 
required  in  order  to  exclude  process  steps  determined  not  applicable  on  a 
project. 

2.3.2. 1  Definitions  of  Figure  2-9  Symbols 

2.3.2. 1.1  "A"  (Line  Management  Heview  and  Approval  i.  This  responsibility  is 
assigned  to  FMSO  Department  iine  managers  tnat  h.r  ■  oeea  tasked  with  the 
development  of  a  Project  or  reaolutu.)  of  a  Program  Trouble  Report  (PTR). 

2.3.2.  1.2  "J"  (Ton  Management  Review  (Optional)).  This  responsibility  is 

assigned  to  a  Project  Review  Board  appointed  bv  tne  Commanding  Officer  to 
review  designated  Command-interest  projects.  The  Commanding  Officer  will  be 
final  approval  authority  on  these  projects. 

2.3.2. 1.3  (Management  Department  (Code  92)  Project  Tracking).  This 

responsibility  is  assigned  to  the  Management  Department  to  administratively 
act  as  FMSO's  front  door  on  all  Project  and  PTR  tasking,  and  to  track  progress 
for  the  Command  via  the  standard  FMSO  project  status  tracking  reporting  system 
of  specific  Command-designated  projects. 

2.3.2. 1.9  "O"  (Management  Department  (Code  92)  Proiect  Management).  This 

responsibility  is  assigned  to  Code  92  for  projects  that  have  significant 
criticai  interfaces  in  two  or  more  Departments  for  which  the  Command  has  not 
specifically  designated  a  Project  Manager.  Project  Managers  will  be  the 
Command  focal  point  for  the  project  and  provide  the  coordination  necessary  to 
ensure  that  all  significant/critical  interfaces  are  resolved. 

2.3.2.  1.5  ^Management  Department  'Code  32)  Quality  Control  fQ/C)). 

This  responsibility  is  assigned  to  the  lanagement  Department  to  assure  that 
all  line  management  tasking  has  been  achieved  within  FMSO  0/A  standards. 
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2. 3. 2.1.6  "0"  (Management  Department  (Code  92)  Quality  Control  (Q/C 

OptionalTH This  responsibility  is  assigned  the  Management  Department  to 
perform  selectively  at  their  discretion  on  designated  development  process 
events . 

2.3.2.  1.7  "CD"  (Management  Department  (Code  92)  tine  Management).  This 
responsibility  is  assigned  to  the  Management  Department  to  perform  line 
management  functions  for  designated  development  process  events  for  all 
projects  where  applicable. 

2. 3. 2.2  Descriptions  of  SDP  Model  Steps 


2. 3.2.2. 1  Tasking  Requirements  Statement  iRS)  or  Proiect.  The  development  of 
a  Requirements  Statement  (formerly  entitled  the  Systems  Policy  and  Concepts 
Statement)  is  the  responsibility  of  the  system  proponent;  however,  current 
Command  policy  is  to  provide  assistance  in  the  preparation  of  the  RS  by  the 
system  proponent  (where  warranted  and  approved  by  the  appropriate  Department 
Director  or  Project  Manager).  The  RS  or  project  tasking  document  will  be 
logged  in  by  Code  92  as  a  Project  Tracking  function  and  forwarded  to  the 
responsible  departments )  for  acceptance  or  rejection. 

2.3.2.2.2  System  Definition  Acceptable  (SYSDEF  OK).  Line  management  will 
review  the  tasking  document  to  ensure  that  it  contains  sufficient  information 
from  which  to  develop  a  functional  description,  cost  benefit  analysis,  plan  of 
action  and  milestones  (POASM)  (internal  or  external),  resource  estimates,  and 
priority  acceptability.  If  sufficient  information  is  not  provided,  a  letter 
citing  tasking  deficiencies  will  be  sent  by  line  management  or  by  the  Project 
Manager  (if  appropriate)  to  NAVSUP  with  a  copy  to  Code  92  to  stop  Project 
Tracking.  Tasking  must  contain  the  general  definition  of  che  target  hardware/ 
software  environment  to  be  used  or  it  must  be  clear  that  an  existing  suite  of 
hardware/software  is  intended.  When  tasking  i*  acceptable  and  the  project  is 

a  new  development,  is  a  new  Application/Operation,  changes  disk  files  or 
teleprocessing,  is  estimated  to  exceed  i.000  manhours  u  FMSO  effort,  or 
may  impact  system  software,  a  copy  of  the  project  will  »e  sent  to  Code  94 
to  provide  estimated  costs  or  leterraine  that  system  software  is  not  affected. 
Code  94  will  respond  to  application  Departments  within  two  working  days  in 
either  case.  When  tasking  is  acceptable  from  all  of  the  above,  line  manage' 
■ent  will  return  a  copy  of  the  project  to  Code  92.  with  total  estimated  costs 
annotated,  for  a  Cost  Senefit  Analysis. 

2. 3.2. 2.3  Cost  Benefit  Analysis.  Code  92  will  develop  a  Cost  Benefit  Analysis 
with  the  assistance  of  line  management.  If  not  cost  beneficial.  Code  92  will 
prepare  a  letter  to  NAVSUP  rejecting  the  project,  update  Project  Tracking 
records,  and  advise  line  management  and  the  Project  Manager  (if  appropriate) 

to  stop  further  effort.  CBA  may  be  subsequently  iterated  at  the  discretion  of 
Code  92  or  line  management. 

2. 3. 2. 2. 4  Estimate  Resources.  Line  management,  including  Code  94  if  involved, 
will  develop  initial  resource  estimates  and  determine  priority  acceptability/ 
required  to  perform  the  tasking.  Resources  include  personnel,  test  bed  and 
operational  hardware,  software,  travel  and  overtime  requirements.  If  there  is 
a  shortfall,  line  management  or  the  Project  Manager  l if  appropriate)  will 
prepare  correspondence  (including  an  impact  statement)  to  NAVSUP  requesting 
additional  resources  or  a  change  in  priority.  A  copy  of  the  letter  will  be 
forwarded  to  Code  92  for  project  Tracking. 
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2 ■ 3 ■ 2 . 2 . 5  P0A4M .  Line  management,  including  Code  94  if  involved  or  the 
Project  Manager  (if  appropriated ,  will  develop  internal  and  external  POA&tts 
for  CO-designated  projects  as  discussed  to  paragraph  4. 1.5. 4. 2.  Exa^les  of 
POA&Ms  are  provided  in  Appendices  4.1-A-l  and  4.1-A-2.  A  copy  of  the  POA&Hs 
will  be  retained  by  Code  92  for  Project  Tracking.  The  CBA,  resource  estimates, 
and  (for  CO-desi gnated  protects)  P0A4H  will  norully  be  done  concurrently  and 
included  in  a  letter  to  NAVSUP  including  a  commitawnt  date  for  FMSO  to  complete 
the  Functional  Description  (FD).  In  addition,  FMSO  line  management  or  the 
Projecc  .Manager  'if  ippropriate)  will  update  external  POA&fs  monthly  for 
submission  to  NAVSl’P  NOTE:  A  senior  executive  Project  Review  Board  tPRB)  has 
been  established  to  execute  FMSO  I  NT  INST  5200. 7B.  Line  management  will,  on 
Commanding  Off icer-desi gnated  projects,  provide  or  present  to  the  PRB  a  System 
Definition  Review  in  iccordance  with  FHSOINTINST  5200. 7B.  When  this  is  approved 
by  the  PRB  and  subsequently  by  the  Commanding  Officer,  line  management  will 
prepare  a  letter  tor  the  Commanding  Officer's  signature  to  NAVSUP  stating  the 
official  FMSO  position/. 

2.J.2.2.0  Approve  PQAoM.  Code  92  will  montor  this  event  as  a  project  track¬ 
ing  responsibility.  When  the  approved  P0A&M  is  received  from  NAVSUP,  the  next 
three  steps  U  e. .  refine  hardware  requirements,  provide  ADS  plan,  provide 
resources/  will  be  initiated  concurrently. 

2. 3. 2. 2.  7  Refine  Hardware  Requireaents.  If  required,  NAVSl/P  will  refine  the 
hardware  requirements  at  a  level  adequate  for  inclusion  m  an  ADS  plan.  Code 
92  will  monitor  this  event  for  progress  as  a  Project  Tracking  task. 

2. 3. 2.2.8  Provide  ADS  Plan.  If  required,  NAVSUP  will  develop  or  update  an 
ADS  plan  and  process  it  up  the  chain  of  coaaand  for  approval.  Although  it  is 
recognized  that  further  FMSO  development  of  the  tasking  should  wait  for  ADS 
plan  approval,  this  nas  proven  to  be  impractical. 

2. 3. 2. 2. 9  Provide  Resources.  If  required,  NAVSUP  will  provide  resources 
and/or  priorities  necessary  to  execute  the  POA&M.  Code  92  will  monitor  this 
event  for  progress  is  i  Project  Tracking  task. 

2.3.2.2.10  Develop  Functional  Description  (FD).  Line  management  will  develop 
the  Functional  description  t Ft))  and  submit  to ^AVSUP  for  approval,  including 
refined  estimates  of  resources  per  paragraph  2. 3. 2.2. 7,  above,  with  a  copy  to 
Code  92  for  Project  Tracking,  Quality  Control,  and  compliance  with  standards. 

Upon  completion  'f  the  FD,  line  management  or  the  Project  Manager  (if  appro¬ 
priate)  will  conduct  j  System  Design  Review.  On  Commanding  Officer-designated 
projects,  the  review  will  be  provided  or  presented  to  the  PRB  in  accordance 
with  FMSCINTINST  3200. 7B.  Code  92  will  provide  or  present  an  updated  CBA  as 
appropriate.  When  approved  by  the  PRB  and  subsequently  by  the  Commanding 
Officer,  *ine  management  or  the  Project  Manager  (if  appropriate)  will  prepare 
a  letter  to  NAVSUP,  for  Commanding  Officer  signature,  including  an  updated 
POA&M  with  a  commitment  late  for  FMSO  to  complete  the  System  Specifications 
iSS). 

2.3.2.2.11  Approve  Functional  Description.  NAVSUP  will  review  the  FD  and 
approve,  approve  with  malif icatioas .  or  disapprove.  This  is  the  critical 
path  to  the  aevelopmeni  of  'he  System  Specification.  NAVSUP  will  update 
resource  requirements  as  required.  Code  92  will  monitor  this  event  for 
progress  as  a  Project  Tracking  task,  if  required. 
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2.3.2.2.12  Acquire  Hardware.  FMSO  assists  by  estimating  capacity  needed  for 

a  representative  site.  NAVSUP  coordinates  with  other  NAVMAT  or  Fleet  claimants, 
performs  data  call  to  all  affected  activities,  and  determines  system-wide 
requirements.  NAVSUP,  directly  or  by  notification  to  other  claimants,  initiates 
acquisition.  Code  92  will  monitor  this  event  for  progress  as  a  Project  Track¬ 
ing  task,  if  required. 

2.3.2.2.13  Develop  System  Specifications  >SS).  Line  management  will  develop 
the  SS  for  release  to  customers  with  a  copy  to  Code  92  for  Project  Tracking 
(if  required).  Quality  Control,  and  standards  review.  In  addition,  at  the 
coupled on  of  the  SS,  line  management  or  the  Project  Manager  (if  appropriate) 
will,  on  CO-designated  projects,  provide  or  present  to  the  PRB  a  Computer 
System  Analysis  Review  in  accordance  with  FMSOINTINST  5200. 7B.  In  addition, 

Code  92  will  provide  or  present  an  updated  C8A  if  appropriate.  When  approved 
by  the  PRB  and  subsequently  by  the  Commanding  Officer,  line  management  or  the 
Project  Manager  (if  appropriate)  will  prepare  a  Letter  to  NAVSUP  for  Commanding 
Officer  signature,  including  an  updated  POA&H  with  a  commitment  Jate  for  FMSO 
to  make  the  program  release. 

2. 3.2.2. 14  Provide  Test  Bed  Hardware.  NAVSUP  provides  hardware  and  system 
software  (if  any)  needed  for  program  development  and  testing.  Code  92  will 
coordinate  or  arrange  the  installation.  Since  this  is  the  critical  path  to 
process  event  2.3.2.2.16,  program  development  can  begin  but  not  be  completed 
if  test  bed  augmentations  or  acquisitions  are  needed  but  not  provided.  Code 

92  will  monitor  this  process  event  on  projects  where  test  bed  hardware/software 
is  required  as  part  of  their  Project  Tracking  function. 

2.3. 2.2.13  Program  Trouble  Report  (?TR).  PTRs  will  be  received  by  Code  92, 
logged  for  PTR  monitoring  as  part  of  their  Project  Tracking  function,  and 
forwarded  to  the  responsible  department  for  resolution.  PTRs  may  affect  any 
development  process  step  in  this  model,  and  are  discussed  in  detail  in  paragraph 

-.2.5. 

2-3.2.2.16  Program  Development,  ,-ine  management  will  develop  Program  Speci¬ 
fications  (PSs),  develop  programs,  perform  unit  testing,  develop  Program 
Maintenance  Manuals  (MMs),  Users  Manuals  (UMs),  and  Computer  Operation  Manuals 
(QMs).  PSs,  UMs,  and  OMs  will  be  released  by  line  management  to  customers. 

Code  92  will  provide  administrative  documentation  release  services  including 
review  of  the  documentation  for  completeness  and  compliance  with  documentation 
and  system  development  process  standards. 

2.3.2.2.17  Develop  Implementation  Plan.  The  customer  is  responsible  for  the 
formulation  of  a  systematic  implementation  plan  based  upon  individual  customer 
requirements.  However,  FMSO  must  assist  the  customer  on  some  projects  by 
developing  a  proposed  plan  and  negotiating  the  issuance  of  a  plan  by  the 
customer.  Negotiations  on  the  implementation  plan  will  be  performed  by  Code 
92  as  a  line  management  function  for  designated  projects,  with  assistance  and 
review/approval  by  line  management  in  affected  departments,  implementation 
plans  required  on  projects  not  designated  for  Code  92  development  will  be 
developed  by  the  appropriate  department  line  management. 

2.3.2.2.18  Testing.  Test  Plans  will  be  developed  and  string  tests  and/or 
system  tests  will  be  performed  by  line  management.  Code  92  will  selectively 
review  test  plana  and  test  requests  for  compliance  with  Quality  Assurance 
standards  and  procedures. 
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2.3.2.2.19  Provide  Hardware  to  Field  Activities.  VAVSUP  and  other  claimants 
will  provide  required  hardware  capacity,  if  any,  for  field  activity  iaplemen- 
ration.  If  required.  Code  92  will  monitor  this  event  for  progress  as  a 
Project  Tracking  function. 

2.3.2.2.20  Prograo  Optimization.  Line  nanageaent  is  routinely  responsible 
for  program  optiaization.  Code  32  will  select  prograas  for  review  and  process¬ 
ing  through  available  optiaization  tools,  and  provide  any  solutions  developed 
to  line  aanagement  by  formal  aeao  with  logic  changes  specified.  Line  manage¬ 
ment  will  schedule  and  modify  the  prograas  in  accordance  with  the  solution 
provided  or  resolve  with  Code  92. 

2.3.2.2.21  Independent  Test  Group.  An  independent  test  group  will  be  estab¬ 
lished  m  Code  92.  For  Code  92-selectea  projects,  entire  release  packages 

will  be  Quality  Controlled  for  compliance  with  standards  and  procedures,  clarity 
and  ease  of  implementation.  Also,  all  output  products  for  the  selected  projects 
will  be  reviewed  for  quality.  In  instances  where  this  effort  will  be  accom¬ 
plished  prior  to  program  release,  line  aanagement  will  be  advised  during 
initial  POA&M  development  for  inclusion  in  estimates.  Recomnwndations  for 
changes  or  corrections  will  be  aade  to  line  aanagement.  Line  aanagement  will 
make  the  changes  or  corrections  in  accordance  with  the  Code  92  recommendations 
or  resolve  with  Code  32. 

2.3. 2 .2. 22  Release  Prograas.  Line  management  will  release  programs  for 
Operational  Review,  Prototype  or  Implementation  when  all  Q/A  functions  have 
been  satisfied.  When  released  for  prototype,  line  msnagement  aay  withhold 
program  releases  to  other  customers  for  implementation  pending  successful 
prototype.  Program  Trouble  Reports  (.PTRs)  or  Flash  notification  will  normally 

be  tot-warded  by  i  prototype  activity  to  FKSO.  Code  92  will  provide  administrative 
release  services  in  accordance  with  current  procedures,  coordinating  the  release 
of  environmental  and  application  software  and  coordinating  resolution  of  hardware 
and  software  interface  requirements.  In  addition,  Code  92  will  review  program 
releases  for  completeness,  clarity  and  compliance  with  documentation  and 
system  development  process  standards  as  a  Code  92  Q/C  function.  If  required. 

Code  92  will  monitor  this  event  for  Project  Management  or  Project  Tracking. 

2.3.2.2.23  OP  Review  or  Prototype.  This  is  the  responsibility  of  the  customer 
and  the  primary  participating  responsibility  of  line  management.  When  this 
occurs.  Code  92  will  participate  at  their  option  as  a  Code  92  Q/C  function. 

If  required.  Code  92  will  monitor  this  function  for  Project  Tracking. 

2.3.2.2.24  Implementation,  [mplement.il  ion  is  a  customer  responsibility  with 
support  provided  by  FMSO.  Support  will  lie  provided  by  line  management  and/or 
Code  92  m  accordance  with  the  implement. mon  plan.  If  required,  Code  92  will 
selectively  monitor  '.his  event  for  Project  Tracking. 

2.3.2.2.25  Post  Release  Review.  As  a  Quality  Control  function,  post  imple¬ 
mentation  visits  will  be  made  to  selected  sites  by  Code  92,  at  their  option, 

to  determine  whether  the  FMSO  program  release  satisfied  the  tasking  and  whether 
the  activity  is  using  t  properly.  Feedback  will  be  provided  to  line  aanagement. 
An  attempt  will  be  node  to  verify  that  the  expected  benefits  were  achieved. 
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APPENDIX  B 

SOFTWARE  RESEARCH  ASSOCIATES  (SRA) 


Software  Research 

ABOUT  SOFTS  ABE  BESEABCH— 

Software  Research  Aasoeizces  (SRA)  is  an  advanced  technology  research  and  engineering 
fire  involved  in  software  science,  software  engineering,  software  duality  assurance,  and 
software  aaincenance.  The  asia  activities  of  the  Company  are  education,  research  and 
develop  a  ent,  consu Icing,  software  cool  design  and  production,  and  allied  technical 
•arvices.  The  Coapeny  hee  offices  in  Sen  Francisco  (headquarters)  end  Los  Angeles, 
California. 

Frofiaeinnel  Develop  aeac  Technology  Saainun— 

The  Coapeny  offers  series  of  Professionel  Develop asne  Seainers  on  a  periodic  basij 
publically,  and  on  an  in -house  beeis  as  well,  SRA  seainers  ere  distinguished  by  their 
dedication  to  preeentetion  of  scate-of-cne-irt  software  engineering  techniques.  Seminar 
offerings  currently  include:  Software  Duality  Aesurance,  Applied  Verification  Tecnnidues. 
Advanced  Software  Validation  Techniques,  Auto  acted  Software  Engineering  Too'. 
Technology,  and  Software  Maintenance  Technology. 

Beaaarcb  and  Develop  ■  ease— 

Company  researchers  tree's  the  latest  technical  developments  in  a  range  of  cr»as. 
including  software  production,  software  testing,  and  software  aeintenar.ee.  as  well  as 
other  areas  of  software  science  and  engineering.  Typical  Coapeny  research  projects 
have  included  work  in  such  areas  ss:  techniques  for  validation  of  software  engineering, 
syste aerie  su-o  nation  of  the  aaintenanee  function,  and  general  methodologies  :cr 
comprehensive  software  tearing  and  analysis. 

Cooeulring  and  Technical  Services— 

Consulting  ‘or  Coapeny  clients  has  ranged  from  evaluation  of  advanced  computer 
architectures  to  the  design  of  states  f-the-srt  software  quality  assurance  rr~an\caticr.». 
The  Company's  approach  to  consulting  emphasizes  complete  technical  disclosure  so  the: 
client  organizations  can  make  enlightened  choices  between  technical  alternatives.  The 
Company  also  provides  specialized  technical  services  using  advanced  software 
engineering  tools.  Such  services  include  software  quality  assurance,  software  testing, 
and  software  maintenance  support. 

Publications— . 

The  Coapany  publiahas  a  quarterly  newsletter,  ‘"Testing  Techniques",  mat  is  distributed 
without  charge  to  qualified  technologists  throughout  the  world.  The  new  newsletter, 
"Quality  Management  Monthly",  is  focused  on  applying  quality  oanagement  techniques 
throughout  Che  software  life  cycle.  The  Company  also  publishes  in  printed  ana 
machinable  form)  the  "Software  Engineering  Automated  Tools  Index"  chat  describes 
soma  500*  software  support  tools. 

Software  Engineering  Tools— 

The  Coapeny  provides  software  production,  testing  and  qualit'-  assurance,  and 
maintenance  tools  for  a  variety  of  computer  svstens.  The  S8TBAR  syscs-t  tf 
structured  programming  preprocessing  orovides  advanced  control  strictures,  internal- : 
prograe  docu  mentation,  ana  jutssatic  inscru  mentation.  The  TCAT  system  for  snrtwar: 
syste  a  teec  coverage  analysis  provides  a  quantitative  Pase  for  qua  lit  assurance  test. nr 
of  C03fih  programs.  The  TTB  interactive  software  analysis  and  testint  tacilf  erjj 
advanced  analvjis  coaeercs  for  support  of  interactive  software  a  iaid  ■  assurer:,.  T:  - 
TSOS  syaten  for  seaantic  update  -,nd  maintenance  of  soilvare  s-.-ote-.s  reorsse-t*  1- 
apvanor  in  the  state  of  the  m  software  confiauricion  sanitecent  ar.f  oro- .tttco 
control. 

Revisec:  Decenber  1961 

»C  &©»  jajt  .  Son  fronosco  .  Ca'r0-,.  »a < .  r*,eonone  at;'  Q*:.iaa-  .  'it,  he  iaC-lli 
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Quali tv  in  a  software  systam  is  a  function  of  logical  integrity  of  evary  part  of  the  syatam 
and  of  tha  systam  as  a  whole.  Verification  (or  'proof")  techniques  are  used  to  help 
establish  tha  needed  levels  of  integrity. 

This  new  SRA  Software  Technology  Seminar  describes  applications  of  tha  "proof  of 
correctness"  methods  to  software  system  quality  control.  In  the  correctness  proving 
approach  conjectures  are  formulated  which  express  correctness  with  respect  to 
specifications.  The  conjectures  are  generated  by  combining  assertions  about  tha  program 
behavior  with  information  from  the  program  source  text.  These  conjectures  are  then  proved 
using  information  about  the  "meaning  of  the  programming  and  specification  languages, 
mathematical  logic,  algebraic  manipulation,  and  mechanical  theorem  proving.  The 
methodology  that  surrounds  the  AFFIRM  system  will  be  described  in  detail. 

This  seminar  is  intended  both  for  individuals  in  R&D  positions  and  software  engineering 
personnel  working  on  highly  reliable  computing  systems.  A  brief  outline  of  the  main  topics 
in  the  seminar  is: 

PHILOSOPHY  AMD  MOTIVATION:  What  is  Verification?;  Programs  as  Mathematical 
Objects;  Unification  of  Verification  and  Design. 

THEORETICAL  FOUNDATIONS:  Inductive  Methods  for  Programs  and  Data;  Proof 
Rile*  tor  Simple  Control  Structures:  .Axiomatic  Specifications  for  Data 
Structures;  State  Transition  Systems;  Foundations  of  the  AFFIRM  Approach; 
Styles  of  Mathematical  Proofs. 

VERIFICATION  METHODS:  Inductive  Assertions;  Recursive  Functions  And  Their 
Proofs;  Proofs  of  Daw  Structure  Properties;  State  Transition  Proofs. 

TECHNOLOGICAL  SUPPORT:  Verification  Conjecture  Generators;  Formula 
Simolifiers,  Rewrite  Rules;  Interactive  Mechanical  Theorem  P rovers;  The 
AFFIRM  Approach. 

SURVEY  OF  APPLICATIONS  OF  VERIFICATION:  Security  Kernels;  Distributed  File 
Systems;  Communication  Protocols. 

The  instructor  for  this  seminar  will  be  DR.  SUSAN  L.  GERHART,  Technical  Director  of 
Software  Research  Associates,  Los  Angeles,  California,  a  post  she  has  held  since  Octooer 
1981.  in  this  capacity  she  is  concerned  primarily  with  the  application  of  verification 
technology  to  practical  problems  of  software  and  system  quality  engineering. 

Dr.  Gerhart  earned  a  BA.  from  Ohio  Wesleyan  University,  a  M.S.  from  the  University  of 
Michigan,  and  a  Ph.D.  from  C  am  egi  e-Mellon  University.  After  serving  on  the  comDuter 
science  faculties  of  the  University  of  Toronto  in  1972-73  and  Duke  University  from  1973  to 
1977.  she  joined  the  Program  Verification  Project  at  USC  Information  Sciences  Institute. 
There  she  participated  in  the  development  of  the  AFFIRM  Specification  and  Verification 
System,  and  served  as  the  AFFIRM  Project  Leader  in  1980-81. 

For  further  information  about  this  and  other  Software  Technology  Seminars  please  check 
the  appropriate  box  on  the  enclosed  Readar  Resprwe  Form  or  call  the  Seminar  Manager  at 
Software  Research  Associates. 

Note:  This  and  other  SRA  seminars  can  be  presented  'in-house'’  to  larger  grouos  of 
attendees  at  substantial  overall  savings  and.  in  most  cases,  partially  tailored  to  a  client’s 
specific  needs.  Please  write  for  a  copy  of  the  SRA  Software  Technology  Seminar  Brochure. 

Software  Research  Aaocutw 

p.o.  Box  :-»3: 

San  Francisco.  CA  ''a  I  la 
Phone.  (41  5)  05"-M4l  —  Telex.  340-335 


As  part  of  its  continuing  research  activity  in  the  Automated  Software  Engineering  Tools 
area.  Software  Research  Associates  has  assembled  a  comprehensive  index  of  detailed  data 
about  a  wide  variety  of  software  engineering  support  tools. 

Available  March  1982,  this  Software  Engineering  Automated  Tools  Index  will  provide 
detailed  information  on  approximately  500  different  software  engineering  tools. 

Tools  described  in  the  Software  ITmiuaanm  Automated  Tools  Index  fall  into  these  major 
categories: 

o  Software  Reqinremants/Speeifleetion  Tools 
o  Software  Design  Tools 

o  Software  Implementation  (Programming)  Tools 
o  Software  Quality  Assurance  Tools 
o  Software  Maintenance  Tools 
o  Software  Project  Management  Toole 


o  Miscellaneous  Utility  Systems 

The  Index  also  includes  a  comprehensive  By-Name  Index,  a  By-Category  Index,  and  a 
complete  BrStggdier  Index.  Available  information  about  obtaining  each  software  system  is 
also  included. 

The  information  in  the  Software  Engineering  Automated  Toole  Index  has  been  gathered  from 
a  wide  range  of  sources  (Government,  Industry,  and  Academia)  over  the  past  three  years. 
Each  automated  tool  is  described  in  a  single  "tool  frame”  that  outlines  such  critical 
information  as  a  tool's  type  and  classification  category,  number  of  installations  and  price, 
special  features  and  exceptional  characteristics,  plus  details  about  the  needed  execution 
environment.  There  are  over  50  tool  categories  divided  equally  among  the  major  system 
classes  mentioned  above. 

The  Software  Engineering  Automated  Tools  Index  is  provided  in  convenient  3-ring  binder 
format,  making  it  easy  to  survey  the  entire  field  of  software  engineering  support  tools,  or 
to  focus  on  just  one  area.  This  format  makes  it  easy  to  incorporate  quarterly  updates  that 
will  be  available  to  current  users  of  the  Software  Engineering  Automated  Tool*  Index.  The 
Two-Volume  Tools  Index  costs  arei  U.S.A./Canada  -  $185.00;  Foreign  -  $225.00.  Costs  for 
the  quarterly  uodates  (available  on  a  subscription  basis)  are:  U.S.A./Canada  •  $85.00; 
Foreign  -  $115.00. 

For  more  information,  or  to  reserve  your  copy  of  the  Index,  please  check  the  appropriate 
boxes  on  the  enclosed  Reader  Response  Form. 

Note:  Machine  processtble  versions  of  the  Software  Engineering  Automated  Tools  Index  are 
also  availaoie  on  special  license  arrangement.  Please  write  SRA  for  details. 

Software  H  aimer  eft  Associates 

P.O.  Sox  2432 
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ADVANCED  SOFTWARE  VALIDATION  TECHNIQUES 

Modern  aethode  of  software  engineering  require  uee  of  advanced  methods  to  as¬ 
sure  the  installed  quality  of  complex  and  critical  software  systeaa. 

This  seainar  addressee  major  issues  facing  the  Verification  and  Validation  com- 
aunity  in  such  areas  as  Symbolic  Evaluation  Methods,  Verification  Methods,  Mu¬ 
tation  Analysis,  Functional  Testing,  Data  Flow  Analysis,  and  Doaain  Testing. 

Besides  describing  how  these  advanced  concepts  can  be  used  in  various  ways  in 
Qualicy  Management  programs,  this  seminar  provides  researchers  and  appliers  of 
these  technologies  with  decailed  information  about  the  payoffs  as  well  as  the 
limitations  of  each  method.  For  example,  should  mutation  analysis  be  done  on 
"large''  programs!  Or,  should  automated  test  data  generation  methods  be  used  in 
a  COBOL  oriented  environment? 

Attendees  will  learn  aboue  state-of-the-art  concepts,  and  will  receive  a 
co^rehensive  set  of  course  notes  and,  in  addition,  a  set  of  reprints  from  the 
current  technical  literature. 


OUTLINES 

STMBOLIC  EXECUTION  TECHNIQUES 
Introduction 

Components  of  a  Symbolic  Execution  System 

Problems  in  Implementing  Symbolic  Execution 

Detection  of  Anomalous  Contracts 

Generation  of  Teat  Data 

Validation  of  Program  Assertions 

Correspondence  Between  Programs  and  Specifications 

Partition  Analysis 

Reliability  of  Synbolic  Execution 

ADVANCES  IN  VERIFICATION 

Definitions 

Verification  by  Case  Analysis 
Inductive  Assertions 
Proofs  with  Symbolic  Evaluation 
Reasoning  from  the  Structure  of  Data 
Practical  Alternatives 

MUTATION  ANALYSIS 

Definition 

Testing  Computer  Programs 
Mutant  Operators 

Relation  to  Other  Testing  Methods 
Practical  Experience 
Systems  That  Have  Been  Built 
Relationship  to  Error  Seeding 

SURVEY  OF  PROMISING  TECHNIQUES 

Functional  Testing 
Data  Flow  Analysis 
Error  Seeding 
Domain  Testing  Strategy 
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THE  INSTRUCTOR 

TIMOTHY  SUOD  it  Aaeiacanc  ?rofeieor  of  Computer  Science  at  the  University  of 
Arizona.  Professor  Sudd ‘a  retearch  intereati  have  foeuaed  on  software  en¬ 
gineering,  program  tearing  and  validation  techniques,  and  high  level  language 
implementation  iaauee.  He  war  a  member  of  the  retearch  team  which  developed 
chc  Program  Mutation  Teating  method,  and  haa  authored  aeveral  papera  on  thia 
and  other  arena  of  program  validation  technology. 

Profeeeor  Sudd  haa  che  Ri.D.  degree  in  Computer  Science  from  Yale  Univereity. 


For  further  information  about  chit  and  other  Software  Technology 
Seminara  please  contact  the  Seminar  Manager  at  Software  Research 
Aaaociatea. . . 


(415)  957-1441 


or  write  to... 


Software  Research  Associates 

- ?.  av"S5T  2m - 


San  Francisco,  California  94126 
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Pall  1981 


SOPTWARE  QUALITY  ASSPRAMCE  TECHHOLOCT 

Developing  procedural  for  aasuring  Chat  a  aoftvart  system  haa  cha  boat  poni- 
bla  chance  to  operate  without  encountering  "buga"  or  “errora"  ia  an  activity 
that  haa  formed  a  najor  focua  of  aoftwara  engineering  technology  for  nearly  a 
decade.  The  goal  of  producing  error-free  aoftwara  reliably  and  efficiently  haa 
eluded  the  beat  theoretical  vorkera,  while  procedurea  for  syatenatically 
analyzing  and  ceaeing  aoftwara  through  atatic  and  dynamic  aitalyaia  haa  gained 
in  popularity,  lecant  development a  in  aoftwara  quality  aaaurance  sake  it  poa- 
sible  to  have  a  reaaonable  expectation  chat  aoftwara  eeeca  minimim  atandarda 
of  coating.  Thia  aaainar  focuaaa  on  tha  concepts,  toola  and  techniques,  con- 
tenporary  reaulti,  and  prognoaia  for  aoftwara  quality  aaauranca  technology. 
Beaidea  providing  an  inveatigation  of  atate-of-tha-art  nethoda  of  program 
atructura  analyaia  (atructurad  taating),  the  aeainar  preaenta  a  variety  of  ma¬ 
terial  that  deala  with  many  altarnativa  phaaaa  of  aoftwara  quality  analyaia. 
Attention  ia  given  not  only  to  the  theoretical  aapecta  of  the  subject  but  alao 
to  practical  reaulta  that  can  likely  be  achieved  by  uae  of  known  methods . 

Attendeea  receive  an  axtenai ve  aet  of  notea  and  a  copy  of  the  tutorial  text 
Software  Teat in«  and  Validation  Tachniauea.  by  Edward  Miller  and  William  5. 
Howden.  Attendeea  will  gain  an  incraaaad  understanding  of  quality  aaaurance 
proceaaea  and  procedurea  and  will  learn  techniques  that  can  be  applied  inedi 
acely  to  quality  assurance  problems. 


OffTLISE: 


IKTRODUCTIOH  AHP  OVERVIEW 

Introduction  to  Methodology 
History  of  Testing  and  QA 
Limits  of  Technology 
Overview  of  Methodology 
Theoretical  Implications /Limit at ions 

MAHAGDffigT  ASPECTS 

Organisational  Setup 
Psychological  Issues 
Level  of  Independence 
Typical  Sasults  of  QA 
Casa  Studias 
Toolset  Description 
Guidelines  and  Limits 

CODE  IHSPECTIOH  AMD  STATIC  ANALYSIS 

Goal  of  Static  Analysis 
Code  Inspection  Procedures 
Typical  Coda  Inspection  Rules 
Role  of  Stacie  Analysara 
Case  Studies 
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TEST  PLAHHIHC  PROCEDURES 

Objectives  of  Teat  Planning 
Role  of  Coverage  Measures 

Structure  of  Progress  (Graph  Theory) 
Pure-Structured  Prograsa '  Teat  Plana 
Hierarchical  Decomposition  Kathode 
Staciaciea  and  Inferencea 

TEST  DATA  SELZCTIOH  METHODS 

Critical  Values  Identified 
Optisua  Choice  of  Specific  Valuea 
Theoretical  Juatificationa 
Relation  to  Proof  of  Correctneaa 
Examples 
Guideline! 

COVERAGE  ANALYSIS 

Heed  for  Coverage  Meaauree 
Cl  Defined  and  Explained 
Ct  Defined  and  Explained 
SI  Defined  and  Explained 
Analyaia  for  Ci/Si  Evaluation 
Sa«ia  in  Graph  Theory 

DOCUHEHTATIOW  AM)  RETESTISG 

Heed  for  Documentation 
Data  to  Keep 

Rateating  (Regreaaion  Testing) 

Change  Control  Syates 
Teat  DocuaMntation  Tools 

CASE  STUDIES 

Role  of  Interactive  Test  Support  Systea 

Saall  Example:  ADD 

Mediua  Exaaple:  KLASS,  LEXICAL 

Large  Exaaple:  FORM 

Statiatica  and  Reliability  laauea 

Kacoasendations 

ACEKPA  POX  RESEARCHERS 


THE  INSTRUCTOR 

EDWARD  P.  KILLER.  JR. ■  is  Technical  Director  of  Software  Reaeerch  Aaaociatea, 
Sen  Francisco.  California,  a  firs  devoted  to  advanced  co^uter  technology  and 
software  applications.  Hia  interests  include  software  engineering  aanagemant, 
software  tatting  technology,  software  aaintenance  -echnology,  automated  tool 
design  and  cosputer  architecture. 
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Dr.  Millar  vat  previously  Director  of  cha  Software  Technology  Cencer,  Science 
Applications,  Inc.,  San  franc isco,  and  Director  of  the  frogram  Validation  Pro¬ 
ject  at  Cenaral  lasearch  Corporation,  Santa  Barbara,  California.  Be  received 
s  BSEX  at  Iowa  State  University  in  1982,  an  M.S.  in  Appliad  Mathematics  at  the 
University  of  Colorado  in  1984,  and  the  n>.D.  at  the  University  of  Maryland  in 
1968  vhare  be  was  an  Instructor  from  198A  to  1988. 

Dr.  Millar  is  a  member  of  the  IEZ2  Computer  Society,  the  ACM,  SIAM  and  several 
honorary  sociatiee.  Be  currently  serves  on  several  technical  conittaes  and 
is  an  Aaaociate  Technical  Editor  of  COKPUTEt  Mags* ins. 


for  further  information  about  this  snd  other  Software  Technology 
Seminars  please  contact  the  Seminar  Manager  at  Software  Research 
Assoc iataa. . . 


(415)  957-1441 

or  write  to. . . 


Software  Research  Associates 
f.  0.  Bo*  243 2 

San  francTsco.  California  94126 
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AUTOMATES  SC^fHAM  ENCIHEERIHG  TOOLS 

The  central  issue  of  software  engineering  lies  in  the  usa  of  automatad  tools 
chat  serve  the  software  engineer  by  amplifying  hit  capabilities.  The  software 
life-cycle  can  be  divided  into  five  phases:  Requirements  Analysis,  Design, 

Implementation  (Programing),  Tasting  (Quality  Assurance),  and  Maintenance. 

Specialized  tools  for  each  area  have  been  found  effective  in  many  applica¬ 
tions,  even  while  extensive  tool-building  research  and  development  continues. 

Contemporary  software  engineering  tools  are  exemplified  by  commercially  avail¬ 
able  tools  that  capture  nearly  every  essential  technical  concept  in  good  tool 
environments.  Ranging  from  single  tools  that  perform  one  important  function 
(like  a  source-language  inatrumentor  system)  to  integrated  sets  of  tools  that 
consolidate  a  variety  of  eloaaly  related  functions,  continued  software  en¬ 
gineering  experience  dictates  the  use  of  good  cools  —  and  in  some  eases  the 
replacement  or  upgrade  of  bad  tools. 

This  seminar  introduces  the  concepts  of  automated  Cools  and  how  they  relate  to 
the  software  engineering  life  cycle,  baaed  on  a  state-of-the-art  survey  of 
contemporary  (comercially  or  publicly  available)  software  engineering  tools. 

Besides  providing  an  in-depth  survey  of  tools  that  apply  in  all  five  areas, 
attention  it  devoted  to  system  production  support  Cools  Chat  aid  in  management 
of  software  development  projects.  Attention  it  also  given  to  estimating  when 
certain  conceptually  important  cools  are  expected  to  be  introduced  in  the 
market  place  in  the  near  future. 

Attendees  receive  an  extensive  set  of  notes  and  a  copy  of  the  tutorial  text 
Automated  Tools  for  Software  Engineering,  by  Edward  Miller.  Attendees  will 
gain  increased  appreciation  for  good  software  tool  design,  an  increased  under¬ 
standing  of  how  tools  interact,  and  a  good  feel  for  the  present  state-of-the- 
art  in  automated  cools. 

OUTLINE: 

PHILOSOPHY  OF  AUTOMATION 

Motivating  Forces 
General  Principles 

Overview  of  Software  Engineering  Phases 
Overview  of  Tool  Role 

TOOLS  FOR  SPECIFICATION/REQUIREMENTS 

Analysis  Tools 
Synches  is  Tools 

Manual  Versus  Automated  Versus  Automatable  Mechodologies 
Contemporary  Spec i f icacions/Requirements  Tools 

TOOLS  FOR  DESIGN  ) 

Principles  of  Design  l. 

Modes  of  Design  Assistance 

Limitations  of  Design  Assistance 

Contemporary  Design/lmpleisentacion  Tools 

Interaction  Between  Tools  and  Che  Operating  Environment 

Racomenda  cions  for  Purchase  /Lease  Decisions 

TOOLS  FOR  PROGRAM  IMPLEMENTATION 

Principles  of  Programing 
Programing  Procedures 
Debugging  Concepts 

Contemporary  Program  Implementation  Tools 

I 
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TOOLS  FOR  OUAIITT  ASSURANCE  AW)  TESTING 

Principle*  of  Prograa  Tatting 
Role  of  Tool*  in  Program  Tatting 
Liaitationt  of  Tool*  Applicable  During  Tetting 
Specific  Exa^las  of  Tatting  Tool* 

Recoamendaciona  for  Purehass/Buy  Decision 

TOOLS  FOR  PROGRAM  MAINTENANCE 

Prineiplaa  of  Software  Maintenance 
Liaitationt  of  Automation  for  Prograa  Maintenance 
Specific  Esaapla  of  Maintenance  Tool* 
Recommendations  for  Purchate/Buy  Decision 


THE  INSTRUCTOR 

TOW AM)  F.  MILLER.  JR.  it  Technical  Director  of  Software  Research  Associates, 
San  Francisco,  CalTTomia,  a  fin*  devoted  to  advanced  computer  technology  and 
software  applications.  Hit  interests  include  software  engineering  aanageaent, 
software  testing  technology,  software  maintenance  technology,  automated  tool 
design  and  computer  architecture. 

Dr.  Miller  was  previously  Director  of  the  Software  Technology  Center,  Science 
Applications,  Inc.,  San  Francisco,  and  Director  of  the  Program  Validation  Pro¬ 
ject  at  General  Research  Corporation,  Santa  Barbara,  California.  He  received 
a  BSEE  at  Iowa  State  University  in  1962,  an  M. S.  in  Applied  Mathematics  at  the 
University  of  Colorado  in  1964,  and  the  Ph.D.  at  the  University  of  Maryland  in 
1968  where  he  was  an  Instructor  from  1964  co  1968. 

Dr.  Miller  is  a  member  of  the  IEEE  Computer  Society,  the  ACM,  SIAM  and  several 
honorary  societies.  He  currently  serves  on  several  technical  coanitteet  and 
is  Associate  Technical  Editor  of  COMPUTER  Magazine. 


For  further  information  about  this  and  other  Software  Technology 
Seminars  please  concact  the  Seminar  Manager  at  Software  Research 
Aasociates. . . 


(415?  957-1441 

or  write  to. . . 


Software  Research  Associates 

J.  0.  Bos  2437 

San  Franc'isc’o, "California  94126 
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0SE»  INTERFACE  DESIGN  psychology 


Deer  Interface  Design,  aa  a  topic  in  iti  own  right,  haa  recently  become  the 
focue  of  lignificant  deeign  efforts.  Aa  the  price/perforaance  curve  of 
hardware  continues  to  show  a  decrease  by  a  factor  of  100  each  10  years,  in¬ 
creasing  aphasia  can  (in  fact,  must)  be  put  on  supporting  user  interactions. 

Aa  a  result,  there  it  increased  recognition  in  the  computer  industry  of  the 
eesential  importance  of  tana  like  "ease-of-Iearning"  and  "ease  of  use." 

This  seainar  covers  the  application  of  selected  information  fro*  the  psychology 
of  learning  and  of  vision  and  tiaa  perception  to  Che  design  of  user/eoaputer 
interfaces. 

Detailed  Case  Studies  of  coaaareial  systeae  will  be  presented.  Video  taped 
deaoastrations  of  these  and  tone  experimental  systeas  will  provide  an  awareness 
and  soae  evaluation  of  the  multitude  of  interaction  techniques,  approaches  and 
devices  that  ara  now  available. 

OUTLINE: 

INTRODUCTION 

Evolution  of  User  I/P  Technology 
Anatoay  of  the  Seainar 
User  I /F  Dimensions 
Information  Processing  Model 
Futuristic  User  I/F  Deao 

LEARNING  THEORIES 

Sequential /Parallel  Acquisition 
Linguistic/Spatial  Materials 
Physiological  Basis  for  Thinking  Styles 

CASE  STUDY  1. 

Graphics  Edicor  Workstation 
Structural  Model  Generation  Application 
Tab  let  /'Menu  Interaction 
Goa  Is /Constraints /Rationale 

HUMAN  MEMORY  CHARACTERISTICS 

Short -Tern/Long -Ter*  Meaory 
Recall  Versus  Recognition 
Spatial/Linguistie  Coding 
Rple  of  Information  Organisation 

VISUAL  PERCEPTION  OVERVIEW 

Light/Space/Color /Tiaa  Sensitivities 
Visual  Organization 
Display  Symbols 

CASE  STUDY  2 

Graphics  Editor  Workstation 
Color  Charts  and  Graphs 
Mouse/Manu  Interaction 
Goa  Is /Constraints /Rationale 
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STRESS  IN  USER/COMPUTER  INTERACTION 

Causes  of  Serass 

What  Can  Be  Dona  to  Reduce 
Examples  in  Computer  Systems 

INTERACTIVITY  AMD  THE  PERCEPTION  OF  TIME 

Osar's  Time  Versus  the  Wall  Clock 

Two  Interaction  Models 

Case  Study  of  a  Database  Interaction 

CASE  STUDY  3  AMD  A 

Desktop  Computer  Line  Editor  Study 
Application  S/W  Study 

Operating  System  Interaction  Demonstration 

TEXT  EDITOR  DEMONSTRATIONS 

I. in# /Chirac ter /Screen  Oriented 
Keyboard/Mouae/Tablat  Device* 

Ease-of 'Learning  Versus  Ease-of-Use 
CoMand  Invocation  Methods 

FUTURE  CONSIDERATIONS 

Spatial  Interfaces 

Voice  Interfaces 

Major  Issues  in  the  Field 

THE  INSTRUCTOR 

DR.  JACK  CRIMES  received  his  Ph.D  from  Iowa  Scats  University  in  Electrical  En¬ 
gineering  and  Computer  Science,  his  M.S.  in  Psychology  and  is  currently  a  doc¬ 
toral  student  in  Applied  Cognitive  Psychology  at  the  University  of  Oregon. 

Since  1971,  he  has  been  employed  at  Tektronix,  Inc.,  in  Beaverton,  Oregon, 
where  he  is  currently  a  manager  of  advanced  development  for  desktop  computers. 

Dr.  Grimes'  research  interests  have  recently  focused  on  understanding  Che  na¬ 
ture  of  user-computer  interaction  from  the  user's  perspective.  Previously,  he 
worked  in  Che  areas  of  computer  architecture,  silicon  technology  and  program¬ 
ming  systems. 

Dr.  Grimes  was  a  participant  in  the  China  Technology  Exchange  Program  in  1979, 
gave  presentations  at  the  Computer  Architecture  Workshop  sponsored  by  Nixdorf 
in  1976  in  West  Germany,  snd  participated  in  the  2nd  USA-Japan  Conference  held 
in  Tokyo  in  1975.  Dr.  Grimes  has  previously  given  a  shorter  version  of  this 
seminar  at  SIGGRAPH  '80  and  '81,  the  Sixth  West  Coast  Co^uter  Faire  and  inter¬ 
nally  at  Tektronix. 


For  further  information  about  this  and  other  Software  Technology  Seminars 
please  contact  Che  Seminar  Manager  at  Software  Research  Associates... 

(415)  957-1441 


or  write  to. . . 


Software  Research  Associates 
2432 

San  Francisco,  Ca 1 i fornia  94126 
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SOFTWARE  MAINTENANCE  TECHNOLOGY 

Software  maintenance  can  often  require  SOX  to  80X  of  the  overall  coats  associ¬ 
ated  with  a  software  system's  life  cycle.  Most  of  the  activities  of  software 
maintenance  involve  detailed  recordkeeping,  incremental  change  to  the  software 
system,  and  analysis  of  the  impact  of  changes. 

Current  technology  for  software  maintenance  is  in  its  infancy.  Technical 
methods  for  analysis  of  complex  and  sophisticated  computer  programs  can  migrate 
from  the  research  and  development  arana  into  practice  only  if  care  is  taken  in 
choosing  the  "right"  algorithms  and  the  "appropriate"  methods  of  controlling 
change.  This  seminar  focuses  on  methods  for  handling  software  maintenance  prob¬ 
lems  that  are  highly  analytical  in  nature,  but  tdiieh  can  have  iasediace  practi¬ 
cal  benefit.  Besides  investigating  various  aspects  of  Che  maintenance  problem, 
the  seminar  presence  methods  of  measuring  and  managing  a  variety  of  software 
suincenance  scenarios. 

Attendees  will  receive  a  comprehensive  annotated  bibliography  of  current 
literature  pertaining  to  software  maintenance  technology,  an  extensive  sec  of 
noces  (including  ease  studies  of  typical  maintenance  situations),  and  reprints 
from  the  currenc  technical  literature. 

OUTLINE: 

INTRODUCTION  AND  OVERVIEW 

Importance  of  Maintenance 
Purposes  of  Maintenance 
Principles  of  Maintenance 

PROBLEMS  OF  MAINTENANCE 

User  Knowledge 

Programmer  Effectiveness,  Availability 
System  quality 
Machine  Requirements 
Environment  Reliability 

PROCRAMMING  ISSUES 

Types  of  Changes  and  Releted  Problems 
Msintenence  Scenerios 

Review  Procedures,  Documentetion  Methods 
Development  Practices  to  Eese  Maintenance  Problems 

METRICS  AND  TESTINC  DURING  MAINTENANCE 

Maintenance  Metrics 
Functional  Testing 
Coverage  Testing 

SOFTWARE  SYSTEM  MANAGEMENT  TECHNOLOGY 

Configuration  Control 
Test  Libraries 
Error  Change  Tracking 

MAINTENANCE  AIDS  AND  TOOLS 

Software  Tools 
Methodologies 


Software  Research  Associates  -l-  Sen  Francisco,  California 


1 


Software  Engineering  Technology  Seminars 


Spring  1982  Series 


HAH AC EM ENT  ISSUES 

Scheduling  for  Maintenance 
Prograner  Motivation 
Manpower  Management 

SUMMARY  AHP  RECOMMENDATIONS 

Overall  Maintenance  Plana 
Researchers'  Agenda 
Bibliography 


THE  INSTRUCTOR 

EDWARD  t.  MILLER.  JR.,  is  Technical  Director  of  Software  Research  Associates, 
San  Francisco,  CalTTomia,  a  firm  devoted  to  advanced  computer  technology  and 
software  applications.  His  interests  include  software  engineering  management, 
software  testing  technology,  software  maintenance  technology,  automated  tool 
design  and  computer  architecture. 

Dr.  Miller  waa  previously  Director  of  the  Software  Technology  Center,  Science 
Applications,  Inc.,  San  Prancisco,  and  Director  of  the  Program  Validation  Pro¬ 
ject  ac  General  Research  Corporation,  Santa  Barbara,  California.  He  received  a 
BSEE  at  Iowa  Scate  University  in  1962,  an  M.S.  in  Applied  Mathematics  at  the 
University  of  Colorado  in  1964,  and  the  Ph.D.  at  the  University  of  Maryland  in 
1968  where  he  was  an  Instructor  from  1964  to  1968. 

Dr.  Miller  is  a  madder  of  the  IEEE  Computer  Society,  the  ACM,  SIAM  and  several 
honorary  societies.  He  currently  serves  on  numerous  technical  eo»ittees  and 
is  Technical  editor  of  COMPUTER  Magazine. 


For  further  information  about  this  and  other  Software  Technology 
Seminars  please  contact  the  Seminar  Manager  at  Software  Research 
Associates. . . 


(415)  957-1441 

or  write  to. . . 


Software  Research  Associates 

- 7.  -57  Ej~2£77 - 

San  Francisco.  California  94126 
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NAME... 

Interactive  Test  Bed  UTB)  for  SRTRAN 
PURPOSE... 

Basic  suooort  of  software  quality  assurance  t trough  systematic  testing,  by 
assisting  the  user  in  achieving  high  values  of  Cl  coverage.  Assistance  is 
provided  by  allowing  the  user  to  alter  global  data  and  analyzing  the 
coverage  of  subsequent  executions.  Capability  to  process  Standard 
SRTRAN  programs. 


SYNOPSIS... 

Basic  capability  for  analv  ing  coverage  results  of  executions-  in  an 
interactive  fashion.  Also  orovided  is  ability  to  alter  data  to  program  so  as 
to  alter  program  flow. 

Version  current' v  available  only  for  Data  General  AOS  environment. 
DESCRIPTION... 

A  free-  tending  ore-rocessor  and  testing  aid  for  interactive  analysis  of 
coverage  and  execution  results  of  SRTRAN  programs  and  subprograms. 

The  system  consists  of  a  SRTRAN  instrumentor,  s  oreorocessor  which 
analyzes  the  data  space  of  the  program,  and  an  interactive  program  which 
is  linked  to  the  specified  test  object.  The  preprocessor  automatically 
generates  subroutines  which  are  used  by  the  testbed  specifically  for  the 
given  teat  object. 

Coverage  and  execution  results  are  reported  when  the  user  asks  for  that 
information. 

SPECIAL  FEATURES... 

The  ITB  svstem  automatically  generates  the  code  it  needs  to  successfully 
test  the  test  object.  There  exist  macros  which  allows  the  user  to  set  uo 
an  1TB  in  a  few  instructions. 

A  trace  feature  is  included  which  allows  the  user  to  follow  execution  of 
the  test  object  ina  segment  by  segment  trace.  This  may  be  turned  on  or 
off  at  wilL 

Commands  entered  interactively  are  automatically  stored  away  so  as  to 
give  the  user  a  complete  record  of  his  session  on  disk.  .Also  available  is 
the  ability  to  use  this  Ghosting1  of  previous  sessions  to  be  the  input  file 
to  another  test  bed  session. 

The  entire  data  space  can  be  saved  at  any  time  during  a  test  bed  session 
for  the  user  to  re-use  iater  in  the  same  session. 


PO.  3c«  .  Son  fnrxate  •  Ca*omkj  9*156  •  Wtpwana  (*i9)  957-  <**i  •  t»t*«  ho.  0*0-109 
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DOCUMENTATION— 

rre  com**  with  •  Reference  Manual. 

SRA  provide*  substantial  related  documentation  on  Software  Quality 
Assurance  and  Software  Maintenance. 

AVAILABILITY... 

The  ITB  system  is  currently  onlv  implemented  on  a  Data  General  AOS 
environment. 

REQUIREMENTS... 

The  system  requires  the  Presence  of  a  FORTRAN  compiler  and  an 
SRTRAN  preprocessor. 


CONTACT— 

Mr.  Thomas  E.  Mapp 
Member  of  Technical  Staff 
Software  Research  Associates 
P.  O.  Bos  2432 

San  Francisco,  CA  94126  USA 
(415)  957-1441 


Updated:  March  1981 
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NAME _ 

COBOL  Test  Coverage  Analysis  Tool  (TCAT/COBOL) 

PURPOSE™ 

TCAT  provides  basic  support  oi  Software  Quality  Assurance  through 
systematic  testing  by  measuring  the  Cl  and  PI  coverage  values  for  series  of 
tests  (Cl  is  the  percentage  of  logical  segments  exercised  and  PI  is  the 
percentage  of  paragraphs  exercised). 

SYNOPSB _ 

TCAT  provides  a  basic  free-standing  capability  for  automatic  instrumentation 
of  programs  to  analyze  and  report  Cl  and  PI  coverage  levels.  TCAT 
processes  ANS  Standard  COBOL  programs,  plus  local  machine  dialect  features 
depending  on  the  system  version  and  host. 

Versions  of  TCAT  are  available  for  IBM,  Univac,  ACOS  (Japan  only),  DEC 
VAX/ VMS,  Data  General  MV78000,  and  ONYX  CS002  (RM-COBOL,  Unix) 
computer  environments. 

DESCRIPTION™ 

TCAT  is  a  free-standing  pre-processor/post-processor  system  for  batch 
oriented  analysis  of  testing  effectiveness  of  COBOL  programs. 

The  COBOL  Test  Coverage  Analysis  Tool  consists  of:  (1)  a  comprehensive 
COBOL  automatic  instrumentor,  1NSTRU,  (2)  a  set  of  run-time  routines  that 
are  loaded  and  executed  with  the  instrumented  COBOL  programs,  called 
RUNTIME,  and.  13)  a  standardized  testing  coverage  analysis  package  called 
COVER. 

The  pre-processing  stage  produces  a  Reference  Listing,  used  to  identify  the 
logical  segments  and  paragraohs  within  the  candidate  COBOL  program,  and 
the  post-execution  stage  of  TCAT  activity  produces  two  forms  of  output:  the 
Coverage  Report  and  the  Not  Hit  report.  These  show  the  percentage  of 
coverage  attained  by  testis)  expressed  in  the  Cl  and  PI  measures.  In 
addition,  the  post-processing  system  generates  a  Histogram  Report  that  shows 
the  proportion  of  times  each  segment  and  paragraph  is  executed. 

Coverage  values  attained  by  tests  of  the  COBOL  program  are  reported  on  a 
per-test,  per-test-group,  or  an  ail-test  cumulative  basis. 

Coverage  reporting  normally  is  defaulted  to  a  predefined  set  of  commonly 
used  formats,  out  can  be  put  completely  under  user  control. 

SPECIAL  FEATURES™ 

The  TCAT  system  can  handle  cumulative  multi-run  tests  bv  storing  standard 
coverage  history  records.  Special  blocking  is  used  to  reduce  the  size  of  the 
intermediate  trace  file.  The  level  of  system  overhead  with  this  method  of 
intermediate  file  storage  is  reasonaDly  low. 
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Ti  e  CCAT  system  can  handle  multiple  entry  C030L  source  modules  as  .veil 
as  COBOL  modules  with  multiple  names. 

The  Reference  Listing  produced  by  the  pre-processor  is  specially  annotated  to 
show  complete  details  of  each  logical  segment  in  the  program.  The  listing 
identifies  the  sense  of  each  logical  predicate  outcome  in  the  COBOL  logic, 
and  provides  statistics  about  the  COBOL  program  that  are  useful  for  test 
module  size  comparisons  and  test  difficulty  estimation. 

Otner  features  include  run-time  settaPle  option  settings. 

DOCUMENTATION™ 

r CAT  is  supplied  with  a  comprehensive  Introduction  and  User's  Guide  plus 
special  installation  support  information  as  appropriate. 

Software  Research  Associates  provides  suostantial  related  documentation  on 
Software  Quality  Assurance  and  Software  Maintenance  in  the  form  of  one-day 
and  two-cay  Professional  Development  Seminars  that  can  be  made  available 
for  presentation  upon  request. 

AVAILABILITY™ 

The  COBOL  TCAT  system  is  available  on  a  single-user  binary  license 
agreement  for  a  variety  of  computer  systems  (see  above). 

Full  documentation,  installation-dependent  information,  and  subscription-type 
maintenance  and  upgrade  service  is  also  provided  with  the  basic  license 
agreement.  Maintenance  and  upgrade  service  after  the  first  year's  use  is  also 
available. 

SYSTEM  REQUIREMENTS™ 

The  TCAT  system  requires  the  presence  of  both  a  COBOL  and  a  FORTRAN 
compiler.  (The  post-processing  phase  of  TCAT  is  implemented  in  a  portaole 
subset  of  FORTRAN.)  In  addition,  during  execution  of  instrumented  programs 
the  TCAT  system  requires  the  use  of  one  serial  file. 

CONTACT™ 

Christopher  Walker  i 

Software  Research  Associates  ' 

P.  O.  Box  2432 

San  Francisco,  CA  94126  USA 

Phone:  (415)  957-1441  -  Telex:  340-235 
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NAME— 


Extenued  BASIC  Validation  Test  Suite 

PURPOSE— 

Validation  of  BASIC  interpreters/ compilers  which  contain 
extensions  similar  to  those  found  in  the  DEC  BASIC-PLUS 
language. 

DESCRIPTION— 

The  Extended  BASIC  Validation  Test  Suite  is  designed  to 
validate  the  syntactic  compatibility  of  a  BASIC 
interpreter/compiler  with  the  DEC  BASIC-PLUS  language  . 

The  test  suite  consists  of  over  200  test  programs  from  the 
MBS  Minimal  3 ASIC  Test  Suite  plus  an  additional  Z50  test 
programs  which  test  the  Extended  BASIC  language  features 
of  DEC  BASIC-PLUS.  The  test  programs  cover  standard 
capaoilities,  end  cases,  and  exceptions  for  the  language 
features. 

The  extensions  to  the  DEC  BASIC-PLUS  language  include 
such  features  as  matrix  functions,  block  I/O,  control  flow 
statements  (WHILE,  REPEAT,  etc),  string  functions,  and 
logical  operators.  All  test  groups  are  shown  below. 

The  output  from  the  tests  are  fully  machine  processible, 
thereby  facilitating  later  regression  testing. 

Software  Research  Associates  can  offer  either  a  complete 
testing  service  for  a  client's  BASIC  interpreter/compiler  or 
the  source  code  only  for  the  Extended  BASIC  Test 
Programs. 

AVAILABILITY— 

The  Extended  BASIC  Validation  Test  Suite  is  currently 
available  for  DEC  BASIC-PLUS  compatible  implementations 
of  BASIC.  A  future  implementation  will  oe  compatible  with 
DG  AOS/VS  BASIC.  SRA  can  also  tailor  a  system  to  a 
client's  specific  language  requirements. 

The  DEC  version  of  the  Extended  BASIC  Test  Suite  is 
priced  at  S3200  for  a  single-user,  single-site  restricted 
source  license. 

CONTACT... 

Mr.  Mark  Opperman 
Software  Research  Associates 
P.  O.  Box  2432 
San  Francisco.  CA  9412S 

(415)  957-1441 
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Extended  BASIC  Validation  Test  Suite  Groups 


Group  Language  Feature 


Number  of 
Programs 


1  Simple  Printing  of  string  constants  1 

2  END  and  STOP  ‘ 

3  PRINT i ng  and  simple  assignment  (LET)  9 

•I  Control  Statements  and  REM  7 

5  Variables  , 


6  Numeric  Constants,  Variables 
”  FOR- NEXT 

3  Arrays 

9  Control  Statements 

10  READ,  DATA  and  RESTORE 


11 

12 

13 

14 

15 


INPUT 

Imp  1 emen t a t i on-supp 1 i ed  Functions 
User-defined  Functions 
Numeric  Expressions 
Miscellaneous  Cheeks 


Minimal  BASIC  Tests  (Subtotal) 


20 

12 

29 

7 

15 


37 

13 

21 

24 


208 


16 

Variables 

17 

Arithmetic  Operators 

18 

Logical  Operators 

19 

String  Operators 

20 

Matrix  Operators 

21 

Mathematical  Functions 

22 

Print  Functions 

23 

String  Functions 

24 

System  Functions 

25 

Matrix  Functions 

26 

Input/Output  Functions 

27 

Extended  Statements 

28 

Matrix  Statements 

29 

Statement  Modifiers 

30 

Block  I/O  Statements 

31 

Miscellaneous  Features 

32 

Inmeoiate  Mode 

16-32 

E:  tended  BASIC  Tests  (Subtotal) 

1-32 

Extended  3AS1C  Test  Suite  (Total 

i 
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SOFTWARE  ENGINEERING  AUTOMATED  TOOLS  OTOD 


PROSPECTUS 


SOFTWARE  ENGINEERING  AUTOMATED  TOOLS  INDEX  —  PROSPECTUS 


TS -875/1 


November  1981 


§  Copyright  1981  by  Software  Research  Aatociacee 

ALL  RIGHTS  RESERVED.  No  part  of  this  document  nay  be  reproduced  in 
any  fora,  by  photostat,  aicrofila,  retrieval  syscea,  or  by  any 
other  aeans  now  known  or  hereafter  invented,  without  written  per¬ 
mission  froa  Software  Research  Associates. 


Software  Research  Associates 
P.  0.  Sox  2432 

San  Francisco,  CA  94126  USA 


Phone:  (413)  937-1441  —  Talar:  340-233 


Software  Research  Associates 


San  Francisco,  California 
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SOFTWARE  ENGINEERING  AUTOMATED  TOOLS  INDEX 


PROSPECTUS 


Th*  Software  Engineering  Autaaeted  Toole  Index  ("TOOLS  INDEX")  deecribee  toae 
600  automated  tools  that  are  available  froa  cosmercial,  governmental,  indus¬ 
trial,  and  other  sources  in  the  United  States  and  elsewhere  in  the  world.  All 
tools  are  categorised  and  cross-referenced  in  detail. 


1.0  CONTENTS 


Following  is  the  structural  contents  of  the  TOOLS  INDEX: 

Table  of  Contents 
1.0  Introduction 


1.1  Organization  of  TOOLS  INDEX 

1.2  Contents  of  Tools  Data  Fraaes 

1.3  Cross-Reference  Listings 

1.4  Updates  and  Corrections 

1.5  Sources  of  Inforaation 


2.0 


3.0 


4.0 


5.0 


Tool  Categories  Listing 
Tool  Naae  Cross-Reference  Listing 
Tool  Category  Cross-Reference  Listing 
Tool  Supplier  Cross-Reference  Listing 


6.0  Supplier  Address  Listing 

7.0  SOFTWARE  ENGINEERING  AUTOMATED  TOOLS  INDEX  DATA  FRAMES  (A-Z) 


8.0  References  and  3ibliography 


2.0  AUTOMATED  TOOL  CATEGORIES 

The  TOOLS  INDEX  is  categorized  based  on  each  Tool's  role  in  the  software 
life  cycle.  The  Tools  are  classified  according  to  a  scheme  chat  provides 
a  special  "category  number"  for  each  major  class  of  Tool. 

Following  are  the  major  categories  used  bv  the  TOOLS  INDEX  (Reference  at¬ 
tached  detailed  listing  -  "Automated  Tool  Categories"): 

-  Requirement/Speeifieation  Tools 

-  Software  Design  Tools 

-  Software  Implementation  Tools 

-  Software  Testing  Tools 

-  Software  Maintenance  Tools 

-  Software  Project  Management  Tools 

-  Language  and  Language  Processing  Systems 

-  Utility  Packages 

-  Miscellaneous  Support  Tools 

-  Research  and  Development  Systems  (Future  Prototypes) 


Software  Research  Assoc iatas  -1- 


San  Francisco,  California 
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SOFTWARE  ENGINEERING  AUTOMATED  TOOLS  INDEX 


PROSPECTUS 


3.0  AUTOMATED  TOOL  CROSS-REFERENCE  LISTINCS 

The  TOOLS  INDEX  provides  a  series  af  cross-reference  listings  Co  aaaiac  in 
locating  specific  Cool  data. 

3.1  Tool  Name  Listing 

Concaina  a  three-field  columnized  daacripeion: 

Tool  Naan  Category  Ratar  Suae  liar  Name 

Listing  ia  alphabetical  by  Tool  NatM. 

3.2  Tool  Category  Liating 

Concaina  a  chrca-fiald  coluanized  deecription: 

Category  Number  Tool  Name  Supplier  Name 

Liating  ia  in  numeric  acquanee  by  Category  Nuaber. 

3.3  Tool  Supplier  Liating 

Concaina  a  three-field  coluamized  deecription: 

Supplier  Naae  Tool  Nana  Category  Number 

Listing  ia  alphabetical  by  Supplier  Nairn. 

3.4  ^°°1  Supplier  Address  Listing 

Is  an  alphabetical  listing,  by  Supplier  Naae,  with  addresses  and 
telephone  nuabers. 


4.0  AUTOMATED  TOOL  DATA 

Tools  are  described  on  single  "Frames”  and  organized  alphabetically  by 
Tool  naae.  (Reference  attached  complete  Frame,  Figure  4.1,  and  actual 
aaagile,  Figure  4-2.) 

The  "Frame"  contains  a  set  of  fields  that  describe  various  features  of  a 
particular  Tool: 


Software  Research  Associates  -2-  San  Francisco,  California 
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SOFTWARE  EMC  I  VEER  INC  AUTOMATED  TOOLS  INDEX 


PROSPECTUS 


FICURE  4-1 :  Content*  of  Automated  Tool  "Fret" 

jw 

Short  name  of  tool  (phrase  describing  tool  use). 

Category 

Tool's  numeric  category  (determined  froa  "Automated  Toots 
Categories"  listing  -  assigned  by  SRA). 

Description 

Short  (one  paragraph)  description  of  whet  the  tool  is  and  what  the 
tool  does. 

Muster  of  Installations 

Number  of  Installations. 

Cost 

The  cost  for  the  system  (including  all  options  and  variations). 
Confituration 

The  configuration  on  which  the  tool  operates. 

Contact 

Company  naae  and  mailing  address  to  contact  about  this  tool. 
Telephone 

Telephone  number  of  person  to  contact  about  this  tool. 

Notes 

Special  notes  about  the  technical  capbilities  and  features  of  this 
particular  tool. 

References 

Any  technical  references  that  describe  how  this  tool  operates,  its 
effectiveness,  or  its  application  (using  standard  bibliographic  ci¬ 
tation  format). 

Source 

The  source  of  the  information  in  the  above  (may  be  altered  by  SRA). 
Updated 

SRA  date  of  latest  revision/update  of  this  block  of  information. 

Software  Research  Associates  -3-  San  Frsneieeo,  California 
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Automated  Software  Engineering  Tools  Index 


:ndex  —  S 


PICURE  4-2 

Stacie  Completed  Tool  "Prams" 


SAME... 

SRTRAN  1  (Base  line) 

CATECOET. . . 

3.4  (Structured  Progressing  Preprocessors) 

DESCRIPTION. . . 

Structured  Programing  Preprocessor  for  FORTRAN  systems. 
NUMBER  Of  INSTALLATIONS. . . 

Approximately  13. 


COST... 


$730  for  perpetual  single-user  binary  license. 

CONFIGURATION. . . 

Portable  to  most  PORTRAN  environments.  SUTRAS  has  been  successfully  in¬ 
stalled  on  IBM,  Onivac,  Data  General,  DEC,  and  CSC  computer  systems. 

CONTACT. . . 

Software  Research  Associates 

P.  0.  Box  2432 

San  Francisco,  CA  94126 


PHONE... 

(415  )  937-1441 


NOTES.  ■ . 

This  is  SRA's  own  structured  programing  preprocessor.  This  "baseline" 
system  includes  the  standard  set  of  Structured  Programing  constructs  such 
as  IF...  ELSE...  ELSE  IF. ..END  IF,  CASE  OF. ..  CASE. ..  CASE  ELSE.. .END  CASE, 
WHILE. . END  WHILE,  REPEAT. .. END,  etc.  In  addition,  SRTRAN  produces  au¬ 
tomatically  indented,  annotated  listings  of  Che  source  programs  it 
processes. 

SRTRAN  is  documented  in  an  extensive  User's  Manual. 

UPDATED. . . 

1  October  1981 
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PROSPECTUS 


5.0  TOOLS  INDEX  UPDATES /CORRECTIONS 


The  TOOLS  INDEX  update a /cor rec c ions /de let iom  will  be  forwarded  to  sub- 
acriber*  on  a  quarterly  baaia.  SRA  i*  continually  modifying  ita  computer- 
ixed  TOOLS  INDEX  file*  in  order  to  reflect  the  noat  current  information 
available. 


6.0  SUBSCRIPTION  RATES 

The  TOOLS  INDEX,  Volume*  I  6  II,  will  be  available  January  1982.  An  Order 
Form  ia  enclosed.  Subscriptions  for  quarterly  TOOLS  INDEX  updates  will  be 
available  on  a  subaeripton  basis  only  at  tha  rates  quoted  below. 

TOOLS  INDEX  QUARTERLY  UPDATES 

2 -Volume  siet 

U. S. A. /Canada  $185.00  U. S. A. /Canada  $  85.00/Yr. 

Foreign  $225.00  Foreign  $11 5.00/Yr. 

IMPORTANT  INFORMATION 


U. S.A. /Canade  orders  shipped  4th  class  book  rate.  Overseas  mail 
shipped  via  sea  mail  (10-12  week  delivery). 

For  priority  shipping  to  U. S.A. /Canada,  or  airmail  service  (2  week 
delivery)  to  foreign  countries,  pleaae  add  the  following  charges: 


Tools  Index: 

Subscription 

Updates 


U. S.A. /Canada 
Foreign 

U.S.A. /Canada 
Foreign 


$10. 00/Set 
$50. 00/Set 

$10. 00/Order /Yr. 
$25. 00 /Order /Yr. 


Tools  Index  price  and  quarterly  subscription  rates  are  subject 
to  change  without  notice. 


Foreign  cheeks  must  be  in  U.S.  Dollars  drawn  on  a  U. S.  bank. 
5.1  Computerized  TOOLS  INDEX 


Computer  readable  versions  of  the  TOOLS  INDEX  are  available  on  special  re¬ 
quest. 


For  further  information  or  ordering  details,  please  contact: 

Ms.  Terryl  Ostao 

Software  Research  Associates 

P.0.  Box  2432 

San  Francisco,  California  94126 
Telephone:  (415)  957-1441  —  Telex:  340-235 
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